"The 3rd generation embedding models from OpenAI (`text-embedding-3-small` and `text-embedding-3-large`) can both be used with our `OpenAIEncoder` and usage is primarily the same as with the 2nd generation `text-embedding-ada-002`. However, there is a new `dimensions` parameter — which we will discuss below."
"Hybrid indexes combine both sparse and dense encodings to produce more accurate results. The dense encoder allows us to search based on semantic meaning, while the sparse encoder allows us to search based on text matches. Merging both dense and sparse into a single hybrid retrieval step allows us to step up our performance beyond what dense-only or sparse-only could achieve."
]
},
{
...
...
@@ -32,7 +32,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We start by installing semantic-router. Support for the new `dimensions` parameter was added in `semantic-router==0.0.19` and `openai==1.10.0`."
"We start by installing semantic-router. Support for the new `AurelioSparseEncoder` parameter was added in `semantic-router==0.1.0`."
"Now we initialize our embedding model, we will use the `-3-large` model alongside a `dimensions` value of `256`. This will produce _tiny_ 256-dimensional vectors that — according to OpenAI — outperform the 1536-dimensional vectors produced by `text-embedding-ada-002`."
"Now we initialize our embedding models. We are going to use a hybrid index which requires both a dense and sparse encoder. For the sparse encoder we will use the pretrained `bm25` model from the Aurelio Platform and OpenAI's `text-embedding-3-small` for the dense encoder.\n",
"\n",
"To get an API key for the Aurelio Platform, we head to the [Aurelio Platform](https://platform.aurelio.ai/settings/api-keys)."
"We now have both our sparse and dense encoders. When using both sparse and dense encoders we need to initialize an index that supports hybrid, such as the `HybridLocalIndex` or `PineconeIndex`."
]
},
{
...
...
@@ -167,7 +180,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we define the `RouteLayer`. When called, the route layer will consume text (a query) and output the category (`Route`) it belongs to — to initialize a `RouteLayer` we need our `encoder` model and a list of `routes`."
"Now we define the `HybridRouter`. When called, the router will consume text (a query) and output the category (`Route`) it belongs to — to initialize a `HybridRouter` we need an `encoder`, `sparse_encoder` our `routes`, and the hybrid `index` we just define."
"2024-11-24 12:25:41 - httpx - INFO - _client.py:1013 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n"
"2024-11-24 18:12:37 - httpx - INFO - _client.py:1013 - _send_single_request() - HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n"
]
},
{
...
...
%% Cell type:markdown id: tags:
[](https://colab.research.google.com/github/aurelio-labs/semantic-router/blob/main/docs/encoders/aurelio-bm25.ipynb) [](https://nbviewer.org/github/aurelio-labs/semantic-router/blob/main/docs/encoders/aurelio-bm25.ipynb)
%% Cell type:markdown id: tags:
# Using Aurelio AI BM25 Encoder
%% Cell type:markdown id: tags:
The 3rd generation embedding models from OpenAI (`text-embedding-3-small` and `text-embedding-3-large`) can both be used with our `OpenAIEncoder` and usage is primarily the same as with the 2nd generation `text-embedding-ada-002`. However, there is a new `dimensions` parameter — which we will discuss below.
Hybrid indexes combine both sparse and dense encodings to produce more accurate results. The dense encoder allows us to search based on semantic meaning, while the sparse encoder allows us to search based on text matches. Merging both dense and sparse into a single hybrid retrieval step allows us to step up our performance beyond what dense-only or sparse-only could achieve.
%% Cell type:markdown id: tags:
## Getting Started
%% Cell type:markdown id: tags:
We start by installing semantic-router. Support for the new `dimensions` parameter was added in `semantic-router==0.0.19` and `openai==1.10.0`.
We start by installing semantic-router. Support for the new `AurelioSparseEncoder` parameter was added in `semantic-router==0.1.0`.
%% Cell type:code id: tags:
``` python
!pipinstall-qU"semantic-router==0.1.0.dev2"
!pipinstall-qUsemantic-router==0.1.0
```
%% Cell type:markdown id: tags:
We start by defining a dictionary mapping routes to example phrases that should trigger those routes.
%% Cell type:code id: tags:
``` python
fromsemantic_routerimportRoute
politics=Route(
name="politics",
utterances=[
"isn't politics the best thing ever",
"why don't you tell me about your political opinions",
"don't you just love the president",
"don't you just hate the president",
"they're going to destroy this country!",
"they will save the country!",
],
)
```
%% Output
/Users/jamesbriggs/Library/Caches/pypoetry/virtualenvs/semantic-router-C1zr4a78-py3.12/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
%% Cell type:markdown id: tags:
Let's define another for good measure:
%% Cell type:code id: tags:
``` python
chitchat=Route(
name="chitchat",
utterances=[
"how's the weather today?",
"how are things going?",
"lovely weather today",
"the weather is horrendous",
"let's go to the chippy",
],
)
routes=[politics,chitchat]
```
%% Cell type:markdown id: tags:
Now we initialize our embedding model, we will use the `-3-large` model alongside a `dimensions` value of `256`. This will produce _tiny_ 256-dimensional vectors that — according to OpenAI — outperform the 1536-dimensional vectors produced by `text-embedding-ada-002`.
Now we initialize our embedding models. We are going to use a hybrid index which requires both a dense and sparse encoder. For the sparse encoder we will use the pretrained `bm25` model from the Aurelio Platform and OpenAI's `text-embedding-3-small` for the dense encoder.
To get an API key for the Aurelio Platform, we head to the [Aurelio Platform](https://platform.aurelio.ai/settings/api-keys).
We now have both our sparse and dense encoders. When using both sparse and dense encoders we need to initialize an index that supports hybrid, such as the `HybridLocalIndex` or `PineconeIndex`.
Now we define the `RouteLayer`. When called, the route layer will consume text (a query) and output the category (`Route`) it belongs to — to initialize a `RouteLayer` we need our `encoder` model and a list of `routes`.
Now we define the `HybridRouter`. When called, the router will consume text (a query) and output the category (`Route`) it belongs to — to initialize a `HybridRouter` we need an `encoder`, `sparse_encoder` our `routes`, and the hybrid `index` we just define.
In this case, we return `None` because no matches were identified. We always recommend optimizing your `RouteLayer` for optimal performance, you can see how in [this notebook](https://github.com/aurelio-labs/semantic-router/blob/main/docs/06-threshold-optimization.ipynb).
