Skip to content
Snippets Groups Projects
Unverified Commit 302fe173 authored by James Briggs's avatar James Briggs Committed by GitHub
Browse files

Merge pull request #135 from aurelio-labs/luca/multi-routes 

feat: Multiple routes added
parents 74f642cb d3d23649
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
docs/00-introduction.ipynb
+ 77
− 19
View file @ 302fe173
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/aurelio-labs/semantic-router/blob/main/docs/00-introduction.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/aurelio-labs/semantic-router/blob/main/docs/00-introduction.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/aurelio-labs/semantic-router/blob/main/docs/00-introduction.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/aurelio-labs/semantic-router/blob/main/docs/00-introduction.ipynb)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Semantic Router Intro # Semantic Router Intro
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The Semantic Router library can be used as a super fast route making layer on top of LLMs. That means rather than waiting on a slow agent to decide what to do, we can use the magic of semantic vector space to make routes. Cutting route making time down from seconds to milliseconds. The Semantic Router library can be used as a super fast route making layer on top of LLMs. That means rather than waiting on a slow agent to decide what to do, we can use the magic of semantic vector space to make routes. Cutting route making time down from seconds to milliseconds.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Getting Started ## Getting Started
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
We start by installing the library: We start by installing the library:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
!pip install -qU semantic-router==0.0.34 !pip install -qU semantic-router==0.0.35
``` ```
%% Output
[notice] A new release of pip is available: 23.1.2 -> 24.0
[notice] To update, run: python.exe -m pip install --upgrade pip
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
We start by defining a dictionary mapping routes to example phrases that should trigger those routes. We start by defining a dictionary mapping routes to example phrases that should trigger those routes.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
from semantic_router import Route from semantic_router import Route
politics = Route( politics = Route(
name="politics", name="politics",
utterances=[ utterances=[
"isn't politics the best thing ever", "isn't politics the best thing ever",
"why don't you tell me about your political opinions", "why don't you tell me about your political opinions",
"don't you just love the president", "don't you just love the president",
"don't you just hate the president", "don't you just hate the president",
"they're going to destroy this country!", "they're going to destroy this country!",
"they will save the country!", "they will save the country!",
], ],
) )
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Let's define another for good measure: Let's define another for good measure:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
chitchat = Route( chitchat = Route(
name="chitchat", name="chitchat",
utterances=[ utterances=[
"how's the weather today?", "how's the weather today?",
"how are things going?", "how are things going?",
"lovely weather today", "lovely weather today",
"the weather is horrendous", "the weather is horrendous",
"let's go to the chippy", "let's go to the chippy",
], ],
) )
routes = [politics, chitchat] routes = [politics, chitchat]
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Now we initialize our embedding model: Now we initialize our embedding model:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import os import os
from getpass import getpass from getpass import getpass
from semantic_router.encoders import CohereEncoder, OpenAIEncoder from semantic_router.encoders import CohereEncoder, OpenAIEncoder
# os.environ["COHERE_API_KEY"] = os.getenv("COHERE_API_KEY") or getpass( # os.environ["COHERE_API_KEY"] = os.getenv("COHERE_API_KEY") or getpass(
# "Enter Cohere API Key: " # "Enter Cohere API Key: "
# ) # )
os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY") or getpass( os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY") or getpass(
"Enter OpenAI API Key: " "Enter OpenAI API Key: "
) )
# encoder = CohereEncoder() # encoder = CohereEncoder()
encoder = OpenAIEncoder() encoder = OpenAIEncoder()
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Now we define the `RouteLayer`. When called, the route layer will consume text (a query) and output the category (`Route`) it belongs to — to initialize a `RouteLayer` we need our `encoder` model and a list of `routes`. Now we define the `RouteLayer`. When called, the route layer will consume text (a query) and output the category (`Route`) it belongs to — to initialize a `RouteLayer` we need our `encoder` model and a list of `routes`.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
from semantic_router.layer import RouteLayer from semantic_router.layer import RouteLayer
rl = RouteLayer(encoder=encoder, routes=routes) rl = RouteLayer(encoder=encoder, routes=routes)
``` ```
%% Output %% Output
2024-01-07 18:08:29 INFO semantic_router.utils.logger Initializing RouteLayer 2024-04-19 18:34:06 INFO semantic_router.utils.logger local
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Now we can test it: Now we can test it:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
rl("don't you love politics?") rl("don't you love politics?")
``` ```
%% Output %% Output
RouteChoice(name='politics', function_call=None) RouteChoice(name='politics', function_call=None, similarity_score=None)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
rl("how's the weather today?") rl("how's the weather today?")
``` ```
%% Output %% Output
RouteChoice(name='chitchat', function_call=None) RouteChoice(name='chitchat', function_call=None, similarity_score=None)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Both are classified accurately, what if we send a query that is unrelated to our existing `Route` objects? Both are classified accurately, what if we send a query that is unrelated to our existing `Route` objects?
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
rl("I'm interested in learning about llama 2") rl("I'm interested in learning about llama 2")
``` ```
%% Output %% Output
RouteChoice(name=None, function_call=None) RouteChoice(name=None, function_call=None, similarity_score=None)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
In this case, we return `None` because no matches were identified. We can also retrieve multiple routes with its associated score:
%% Cell type:code id: tags:
``` python
rl.retrieve_multiple_routes("Hi! How are you doing in politics??")
```
%% Output
[RouteChoice(name='politics', function_call=None, similarity_score=0.8596186767854487),
RouteChoice(name='chitchat', function_call=None, similarity_score=0.8356239688161808)]
%% Cell type:code id: tags:
``` python
rl.retrieve_multiple_routes("I'm interested in learning about llama 2")
```
%% Output
[]
%% Cell type:code id: tags:
``` python
```
......
semantic_router/layer.py
+ 70
− 8
View file @ 302fe173
...@@ -271,6 +271,32 @@ class RouteLayer: ...@@ -271,6 +271,32 @@ class RouteLayer:
# if no route passes threshold, return empty route choice # if no route passes threshold, return empty route choice
return RouteChoice() return RouteChoice()
def retrieve_multiple_routes(
self,
text: Optional[str] = None,
vector: Optional[List[float]] = None,
) -> List[RouteChoice]:
if vector is None:
if text is None:
raise ValueError("Either text or vector must be provided")
vector_arr = self._encode(text=text)
else:
vector_arr = np.array(vector)
# get relevant utterances
results = self._retrieve(xq=vector_arr)
# decide most relevant routes
categories_with_scores = self._semantic_classify_multiple_routes(results)
route_choices = []
for category, score in categories_with_scores:
route = self.check_for_matching_routes(category)
if route:
route_choice = RouteChoice(name=route.name, similarity_score=score)
route_choices.append(route_choice)
return route_choices
def _retrieve_top_route( def _retrieve_top_route(
self, vector: List[float], route_filter: Optional[List[str]] = None self, vector: List[float], route_filter: Optional[List[str]] = None
) -> Tuple[Optional[Route], List[float]]: ) -> Tuple[Optional[Route], List[float]]:
...@@ -423,14 +449,7 @@ class RouteLayer: ...@@ -423,14 +449,7 @@ class RouteLayer:
) )
def _semantic_classify(self, query_results: List[dict]) -> Tuple[str, List[float]]: def _semantic_classify(self, query_results: List[dict]) -> Tuple[str, List[float]]:
scores_by_class: Dict[str, List[float]] = {} scores_by_class = self.group_scores_by_class(query_results)
for result in query_results:
score = result["score"]
route = result["route"]
if route in scores_by_class:
scores_by_class[route].append(score)
else:
scores_by_class[route] = [score]
# Calculate total score for each class # Calculate total score for each class
total_scores = { total_scores = {
...@@ -446,6 +465,49 @@ class RouteLayer: ...@@ -446,6 +465,49 @@ class RouteLayer:
logger.warning("No classification found for semantic classifier.") logger.warning("No classification found for semantic classifier.")
return "", [] return "", []
def get(self, name: str) -> Optional[Route]:
for route in self.routes:
if route.name == name:
return route
logger.error(f"Route `{name}` not found")
return None
def _semantic_classify_multiple_routes(
self, query_results: List[dict]
) -> List[Tuple[str, float]]:
scores_by_class = self.group_scores_by_class(query_results)
# Filter classes based on threshold and find max score for each
classes_above_threshold = []
for route_name, scores in scores_by_class.items():
# Use the get method to find the Route object by its name
route_obj = self.get(route_name)
if route_obj is not None:
# Use the Route object's threshold if it exists, otherwise use the provided threshold
_threshold = (
route_obj.score_threshold
if route_obj.score_threshold is not None
else self.score_threshold
)
if self._pass_threshold(scores, _threshold):
max_score = max(scores)
classes_above_threshold.append((route_name, max_score))
return classes_above_threshold
def group_scores_by_class(
self, query_results: List[dict]
) -> Dict[str, List[float]]:
scores_by_class: Dict[str, List[float]] = {}
for result in query_results:
score = result["score"]
route = result["route"]
if route in scores_by_class:
scores_by_class[route].append(score)
else:
scores_by_class[route] = [score]
return scores_by_class
def _pass_threshold(self, scores: List[float], threshold: float) -> bool: def _pass_threshold(self, scores: List[float], threshold: float) -> bool:
if scores: if scores:
return max(scores) > threshold return max(scores) > threshold
......
tests/unit/test_layer.py
+ 159
− 0
View file @ 302fe173
...@@ -97,6 +97,22 @@ def routes(): ...@@ -97,6 +97,22 @@ def routes():
] ]
@pytest.fixture
def routes_2():
return [
Route(name="Route 1", utterances=["Hello"]),
Route(name="Route 2", utterances=["Hello"]),
]
@pytest.fixture
def routes_3():
return [
Route(name="Route 1", utterances=["Hello"]),
Route(name="Route 2", utterances=["Asparagus"]),
]
@pytest.fixture @pytest.fixture
def dynamic_routes(): def dynamic_routes():
return [ return [
...@@ -503,6 +519,149 @@ class TestRouteLayer: ...@@ -503,6 +519,149 @@ class TestRouteLayer:
) )
assert route_layer.get_thresholds() == {"Route 1": 0.82, "Route 2": 0.82} assert route_layer.get_thresholds() == {"Route 1": 0.82, "Route 2": 0.82}
def test_with_multiple_routes_passing_threshold(
self, openai_encoder, routes, index_cls
):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes, index=index_cls()
)
route_layer.score_threshold = 0.5 # Set the score_threshold if needed
# Assuming route_layer is already set up with routes "Route 1" and "Route 2"
query_results = [
{"route": "Route 1", "score": 0.6},
{"route": "Route 2", "score": 0.7},
{"route": "Route 1", "score": 0.8},
]
# Override _pass_threshold to always return True for this test
route_layer._pass_threshold = lambda scores, threshold: True
expected = [("Route 1", 0.8), ("Route 2", 0.7)]
results = route_layer._semantic_classify_multiple_routes(query_results)
assert sorted(results) == sorted(
expected
), "Should classify and return routes above their thresholds"
def test_with_no_routes_passing_threshold(self, openai_encoder, routes, index_cls):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes, index=index_cls()
)
route_layer.score_threshold = 0.5
# Override _pass_threshold to always return False for this test
route_layer._pass_threshold = lambda scores, threshold: False
query_results = [
{"route": "Route 1", "score": 0.3},
{"route": "Route 2", "score": 0.2},
]
expected = []
results = route_layer._semantic_classify_multiple_routes(query_results)
assert (
results == expected
), "Should return an empty list when no routes pass their thresholds"
def test_with_no_query_results(self, openai_encoder, routes, index_cls):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes, index=index_cls()
)
route_layer.score_threshold = 0.5
query_results = []
expected = []
results = route_layer._semantic_classify_multiple_routes(query_results)
assert (
results == expected
), "Should return an empty list when there are no query results"
def test_with_unrecognized_route(self, openai_encoder, routes, index_cls):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes, index=index_cls()
)
route_layer.score_threshold = 0.5
# Test with a route name that does not exist in the route_layer's routes
query_results = [{"route": "UnrecognizedRoute", "score": 0.9}]
expected = []
results = route_layer._semantic_classify_multiple_routes(query_results)
assert results == expected, "Should ignore and not return unrecognized routes"
def test_retrieve_with_text(self, openai_encoder, routes, index_cls):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes, index=index_cls()
)
text = "Hello"
results = route_layer.retrieve_multiple_routes(text=text)
assert len(results) >= 1, "Expected at least one result"
assert any(
result.name in ["Route 1", "Route 2"] for result in results
), "Expected the result to be either 'Route 1' or 'Route 2'"
def test_retrieve_with_vector(self, openai_encoder, routes, index_cls):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes, index=index_cls()
)
vector = [0.1, 0.2, 0.3]
results = route_layer.retrieve_multiple_routes(vector=vector)
assert len(results) >= 1, "Expected at least one result"
assert any(
result.name in ["Route 1", "Route 2"] for result in results
), "Expected the result to be either 'Route 1' or 'Route 2'"
def test_retrieve_without_text_or_vector(self, openai_encoder, routes, index_cls):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes, index=index_cls()
)
with pytest.raises(ValueError, match="Either text or vector must be provided"):
route_layer.retrieve_multiple_routes()
def test_retrieve_no_matches(self, openai_encoder, routes, index_cls):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes, index=index_cls()
)
text = "Asparagus"
results = route_layer.retrieve_multiple_routes(text=text)
assert len(results) == 0, f"Expected no results, but got {len(results)}"
def test_retrieve_one_match(self, openai_encoder, routes_3, index_cls):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes_3, index=index_cls()
)
text = "Hello"
results = route_layer.retrieve_multiple_routes(text=text)
assert len(results) == 1, f"Expected one result, and got {len(results)}"
matched_routes = [result.name for result in results]
assert "Route 1" in matched_routes, "Expected 'Route 1' to be a match"
def test_retrieve_with_text_for_multiple_matches(
self, openai_encoder, routes_2, index_cls
):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes_2, index=index_cls()
)
text = "Hello"
results = route_layer.retrieve_multiple_routes(text=text)
assert len(results) == 2, "Expected two results"
matched_routes = [result.name for result in results]
assert "Route 1" in matched_routes, "Expected 'Route 1' to be a match"
assert "Route 2" in matched_routes, "Expected 'Route 2' to be a match"
def test_set_aggregation_method_with_unsupported_value(
self, openai_encoder, routes, index_cls
):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes, index=index_cls()
)
unsupported_aggregation = "unsupported_aggregation_method"
with pytest.raises(
ValueError,
match=f"Unsupported aggregation method chosen: {unsupported_aggregation}. Choose either 'SUM', 'MEAN', or 'MAX'.",
):
route_layer._set_aggregation_method(unsupported_aggregation)
def test_refresh_routes_not_implemented(self, openai_encoder, routes, index_cls):
route_layer = RouteLayer(
encoder=openai_encoder, routes=routes, index=index_cls()
)
with pytest.raises(
NotImplementedError, match="This method has not yet been implemented."
):
route_layer._refresh_routes()
class TestLayerFit: class TestLayerFit:
def test_eval(self, openai_encoder, routes, test_data): def test_eval(self, openai_encoder, routes, test_data):
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

:feet::heart: Strive to be the person your dogs believe you are :heart::feet: