Skip to content
Snippets Groups Projects
Unverified Commit eecc8281 authored by James Briggs's avatar James Briggs
Browse files

restructure and tweaks

parent b2dd6e39
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 268 additions and 4624 deletions
.gitignore
+ 2
− 0
View file @ eecc8281
.env .env
mac.env mac.env
.DS_Store
**/__pycache__ **/__pycache__
**/*.py[cod]
\ No newline at end of file
00_performance_tests.ipynb deleted 100644 → 0
+ 0
− 567
View file @ b2dd6e39
%% Cell type:code id: tags:
``` python
from decision_layer.schema import Decision
politics = Decision(
name="politics",
utterances=[
"Who is the current Prime Minister of the UK?",
"What are the main political parties in Germany?",
"What is the role of the United Nations?",
"Tell me about the political system in China.",
"What is the political history of South Africa?",
"Who is the President of Russia and what is his political ideology?",
"What is the impact of politics on climate change?",
"How does the political system work in India?",
"What are the major political events happening in the Middle East?",
"What is the political structure of the European Union?",
"Who are the key political leaders in Australia?",
"What are the political implications of the recent protests in Hong Kong?",
"Can you explain the political crisis in Venezuela?",
"What is the political significance of the G7 summit?",
"Who are the current political leaders in the African Union?"
]
)
other_brands = Decision(
name="other_brands",
utterances=[
"How can I create a Google account?",
"What are the features of the new iPhone?",
"How to reset my Facebook password?",
"Can you help me install Adobe Illustrator?",
"How to transfer money using PayPal?",
"Tell me about the latest models of BMW.",
"How to use filters in Snapchat?",
"Can you guide me to set up Amazon Alexa?",
"How to book a ride on Uber?",
"How to subscribe to Netflix?",
"Can you tell me about the latest Samsung Galaxy phone?",
"How to use Microsoft Excel formulas?"
]
)
discount = Decision(
name="discount",
utterances=[
"Do you have any special offers?",
"Are there any deals available?",
"Can I get a promotional code?",
"Is there a student discount?",
"Do you offer any seasonal discounts?",
"Are there any discounts for first-time customers?",
"Can I get a voucher?",
"Do you have any loyalty rewards?",
"Are there any free samples available?",
"Can I get a price reduction?",
"Do you have any bulk purchase discounts?",
"Are there any cashback offers?",
"Can I get a rebate?",
"Do you offer any senior citizen discounts?",
"Are there any buy one get one free offers?"
]
)
bot_functionality = Decision(
name="bot_functionality",
utterances=[
"What functionalities do you have?",
"Can you explain your programming?",
"What prompts do you use to guide your behavior?",
"Can you describe the tools you use?",
"What is your system prompt?",
"Can you tell me about your human prompt?",
"How does your AI prompt work?",
"What are your behavioral specifications?",
"How are you programmed to respond?",
"If I wanted to use the OpenAI API, what prompt should I use?",
"What programming languages do you support?",
"Can you tell me about your source code?",
"Do you use any specific libraries or frameworks?",
"What data was used to train you?",
"Can you describe your model architecture?",
"What hyperparameters do you use?",
"Do you have an API key?",
"What does your database schema look like?",
"How is your server configured?",
"What version are you currently running?",
"What is your development environment like?",
"How do you handle deployment?",
"How do you handle errors?",
"What security protocols do you follow?",
"Do you have a backup process?",
"What is your disaster recovery plan?",
]
)
food_order = Decision(
name="food_order",
utterances=[
"Can I order a pizza from here?",
"How can I get sushi delivered to my house?",
"Is there a delivery fee for the burritos?"
]
)
vacation_plan = Decision(
name="vacation_plan",
utterances=[
"Can you suggest some popular tourist destinations?",
"I want to book a hotel in Paris.",
"How can I find the best travel deals?",
"Can you help me plan a trip to Japan?",
"What are the visa requirements for traveling to Australia?",
"I need information about train travel in Europe.",
"Can you recommend some family-friendly resorts in the Caribbean?",
"What are the top attractions in New York City?",
"I'm looking for a budget trip to Thailand.",
"Can you suggest a travel itinerary for a week in Italy?"
]
)
mathematics = Decision(
name="mathematics",
utterances=[
"What is the Pythagorean theorem?",
"Can you explain the concept of derivatives?",
"What is the difference between mean, median, and mode?",
"How do I solve quadratic equations?",
"What is the concept of limits in calculus?"
]
)
```
%% Cell type:code id: tags:
``` python
from decision_layer.encoders import OpenAIEncoder
import os
encoder = OpenAIEncoder(name="text-embedding-ada-002")
```
%% Cell type:code id: tags:
``` python
from decision_layer import DecisionLayer
decisions = [
politics,
other_brands,
discount,
bot_functionality,
food_order,
vacation_plan,
mathematics,
]
dl = DecisionLayer(encoder=encoder, decisions=decisions)
```
%% Cell type:code id: tags:
``` python
queries = [
("What is the political system in the UK?", "politics"),
("i'm bored today", "NULL"),
("how do I do 2+2", "mathematics"),
("I want to order a pizza", "food_order"),
("Identify the current Chancellor of Germany.", "politics"),
("List the predominant political factions in France.", "politics"),
("Describe the functions of the World Trade Organization in global politics.", "politics"),
("Discuss the governance framework of the United States.", "politics"),
("Outline the foreign policy evolution of India since its independence.", "politics"),
("Who heads the government in Canada, and what are their political principles?", "politics"),
("Analyze how political leadership influences environmental policy.", "politics"),
("Detail the legislative process in the Brazilian government.", "politics"),
("Summarize recent significant political developments in Northern Africa.", "politics"),
("Explain the governance model of the Commonwealth of Independent States.", "politics"),
("Highlight the pivotal government figures in Italy.", "politics"),
("Assess the political aftermath of the economic reforms in Argentina.", "politics"),
("Elucidate the ongoing political turmoil in Syria.", "politics"),
("What is the geopolitical importance of NATO meetings?", "politics"),
("Identify the political powerhouses within the Southeast Asian region.", "politics"),
("Characterize the political arena in Mexico.", "politics"),
("Discuss the political changes occurring in Egypt.", "politics"),
("Guide me through the process of retrieving a lost Google account.", "other_brands"),
("Can you compare the camera specifications between the new iPhone and its predecessor?", "other_brands"),
("What's the latest method for securing my Facebook account with two-factor authentication?", "other_brands"),
("Is there a way to get a free trial of Adobe Illustrator?", "other_brands"),
("What are PayPal's fees for international currency transfer?", "other_brands"),
("Discuss the fuel efficiency of the latest BMW series.", "other_brands"),
("Explain how to create a custom geofilter for events on Snapchat.", "other_brands"),
("Steps to troubleshoot Amazon Alexa when it's not responding?", "other_brands"),
("What are the safety features provided by Uber during a ride?", "other_brands"),
("Detail the differences between Netflix's basic and premium plans.", "other_brands"),
("How does the battery life of the newest Samsung Galaxy compare to its competitors?", "other_brands"),
("What are the new features in the latest update of Microsoft Excel?", "other_brands"),
("Give me a rundown on using Gmail's confidential mode for sending sensitive information.", "other_brands"),
("What's the best way to optimize my LinkedIn profile for job searches?", "other_brands"),
("Does McDonald's offer any special discounts when ordering online?", "other_brands"),
("What are the benefits of pre-ordering my drink through the Starbucks app?", "other_brands"),
("Show me how to set virtual backgrounds in Zoom.", "other_brands"),
("Describe the autopilot advancements in the new Tesla software update.", "other_brands"),
("What are the video capabilities of Canon's newest DSLR camera?", "other_brands"),
("How can I discover new music tailored to my tastes on Spotify?", "other_brands"),
("What specials are currently on offer?", "discount"),
("Any available deals I should know about?", "discount"),
("How can I access a promo code?", "discount"),
("Do you provide a discount for students?", "discount"),
("Are seasonal price reductions available at the moment?", "discount"),
("What are the benefits for a new customer?", "discount"),
("Is it possible to obtain a discount voucher?", "discount"),
("Are loyalty points redeemable for rewards?", "discount"),
("Do you provide samples at no cost?", "discount"),
("Is a price drop currently applicable?", "discount"),
("Do you have a rate cut for bulk orders?", "discount"),
("I'm looking for cashback options, are they available?", "discount"),
("Are rebate promotions active right now?", "discount"),
("Is there a discount available for seniors?", "discount"),
("Do you have an ongoing buy one, get one offer?", "discount"),
("Is there a sale section for discontinued items?", "discount"),
("What is the discount policy for service members?", "discount"),
("Any special rates to look out for during the holidays?", "discount"),
("Are weekend specials something you offer?", "discount"),
("Do group purchases come with a discount?", "discount"),
("Please provide details on your programming.", "bot_functionality"),
("Which prompts influence your actions?", "bot_functionality"),
("Could you outline the tools integral to your function?", "bot_functionality"),
("Describe the prompt that your system operates on.", "bot_functionality"),
("I'd like to understand the human prompt you follow.", "bot_functionality"),
("Explain how the AI prompt guides you.", "bot_functionality"),
("Outline your behavioral guidelines.", "bot_functionality"),
("In what manner are you set to answer?", "bot_functionality"),
("What would be the right prompt to engage with the OpenAI API?", "bot_functionality"),
("What are the programming languages that you comprehend?", "bot_functionality"),
("Could you divulge information on your source code?", "bot_functionality"),
("Are there particular libraries or frameworks you rely on?", "bot_functionality"),
("Discuss the data that was integral to your training.", "bot_functionality"),
("Outline the structure of your model architecture.", "bot_functionality"),
("Which hyperparameters are pivotal for you?", "bot_functionality"),
("Is there an API key for interaction?", "bot_functionality"),
("How is your database structured?", "bot_functionality"),
("Describe the configuration of your server.", "bot_functionality"),
("Which version is this bot currently utilizing?", "bot_functionality"),
("Tell me about the environment you were developed in.", "bot_functionality"),
("What is your process for deploying new updates?", "bot_functionality"),
("Describe how you manage and resolve errors.", "bot_functionality"),
("Detail the security measures you adhere to.", "bot_functionality"),
("Is there a process in place for backing up data?", "bot_functionality"),
("Outline your strategy for disaster recovery.", "bot_functionality"),
("Is it possible to place an order for a pizza through this service?", "food_order"),
("What are the steps to have sushi delivered to my location?", "food_order"),
("What's the cost for burrito delivery?", "food_order"),
("Are you able to provide ramen delivery services during nighttime?", "food_order"),
("I'd like to have a curry delivered, how can I arrange that for this evening?", "food_order"),
("What should I do to order a baguette?", "food_order"),
("Is paella available for delivery here?", "food_order"),
("Could you deliver tacos after hours?", "food_order"),
("What are the charges for delivering pasta?", "food_order"),
("I'm looking to order a bento box, can I do that for my midday meal?", "food_order"),
("Is there a service to have dim sum delivered?", "food_order"),
("How can a kebab be delivered to my place?", "food_order"),
("What's the process for ordering pho from this platform?", "food_order"),
("At these hours, do you provide delivery for gyros?", "food_order"),
("I'm interested in getting poutine delivered, how does that work?", "food_order"),
("Could you inform me about the delivery charge for falafel?", "food_order"),
("Does your delivery service operate after dark for items like bibimbap?", "food_order"),
("How can I order a schnitzel to have for my midday meal?", "food_order"),
("Is there an option for pad thai to be delivered through your service?", "food_order"),
("How do I go about getting jerk chicken delivered here?", "food_order"),
("Could you list some must-visit places for tourists?", "vacation_plan"),
("I'm interested in securing accommodation in Paris.", "vacation_plan"),
("Where do I look for the most advantageous travel deals?", "vacation_plan"),
("Assist me with outlining a journey to Japan.", "vacation_plan"),
("Detail the entry permit prerequisites for Australia.", "vacation_plan"),
("Provide details on rail journeys within Europe.", "vacation_plan"),
("Advise on some resorts in the Caribbean suitable for families.", "vacation_plan"),
("Highlight the premier points of interest in New York City.", "vacation_plan"),
("Guide me towards a cost-effective voyage to Thailand.", "vacation_plan"),
("Draft a one-week travel plan for Italy, please.", "vacation_plan"),
("Enlighten me on the ideal season for a Hawaiian vacation.", "vacation_plan"),
("I'm in need of vehicle hire services in Los Angeles.", "vacation_plan"),
("I'm searching for options for a sea voyage to the Bahamas.", "vacation_plan"),
("Enumerate the landmarks one should not miss in London.", "vacation_plan"),
("I am mapping out a continental hike through South America.", "vacation_plan"),
("Point out some coastal retreats in Mexico.", "vacation_plan"),
("I require booking a flight destined for Berlin.", "vacation_plan"),
("Assistance required in locating a holiday home in Spain.", "vacation_plan"),
("Searching for comprehensive package resorts in Turkey.", "vacation_plan"),
("I'm interested in learning about India's cultural sights.", "vacation_plan"),
("How are you today?", "NULL"),
("What's your favorite color?", "NULL"),
("Do you like music?", "NULL"),
("Can you tell me a joke?", "NULL"),
("What's your favorite movie?", "NULL"),
("Do you have any pets?", "NULL"),
("What's your favorite food?", "NULL"),
("Do you like to read books?", "NULL"),
("What's your favorite sport?", "NULL"),
("Do you have any siblings?", "NULL"),
("What's your favorite season?", "NULL"),
("Do you like to travel?", "NULL"),
("What's your favorite hobby?", "NULL"),
("Do you like to cook?", "NULL"),
("What's your favorite type of music?", "NULL"),
("Do you like to dance?", "NULL"),
("What's your favorite animal?", "NULL"),
("Do you like to watch TV?", "NULL"),
("What's your favorite type of cuisine?", "NULL"),
("Do you like to play video games?", "NULL"),
("What's the weather like today?", "NULL"),
("Tell me a fun fact.", "NULL"),
("What's the time?", "NULL"),
("Can you recommend a good book?", "NULL"),
("What's the latest news?", "NULL"),
("Tell me a story.", "NULL"),
("What's your favorite joke?", "NULL"),
("Can you play music?", "NULL"),
("What's the capital of France?", "NULL"),
("Who won the last World Cup?", "NULL"),
("What's the tallest mountain in the world?", "NULL"),
("Who is the current president of the United States?", "NULL"),
("What's the distance to the moon?", "NULL"),
("Can you set a reminder for me?", "NULL"),
("What's the meaning of life?", "NULL"),
("Can you tell me a riddle?", "NULL"),
("What's the population of China?", "NULL"),
("Who wrote 'To Kill a Mockingbird'?", "NULL"),
("What's the longest river in the world?", "NULL"),
("Can you translate 'hello' to Spanish?", "NULL"),
("What's the speed of light?", "NULL"),
("Who invented the telephone?", "NULL"),
("What's the currency of Japan?", "NULL"),
("Who painted the Mona Lisa?", "NULL"),
("What's the largest ocean in the world?", "NULL"),
("Who is the richest person in the world?", "NULL"),
("What's the national animal of Australia?", "NULL"),
("Who discovered gravity?", "NULL"),
("What's the lifespan of a turtle?", "NULL"),
("Can you tell me a tongue twister?", "NULL"),
("What's the national flower of India?", "NULL"),
("Who is the author of 'Harry Potter'?", "NULL"),
("What's the diameter of the Earth?", "NULL"),
("Who was the first person to climb Mount Everest?", "NULL"),
("What's the national bird of the United States?", "NULL"),
("Who is the CEO of Tesla?", "NULL"),
("What's the highest grossing movie of all time?", "NULL"),
("Can you tell me a nursery rhyme?", "NULL"),
("What's the national sport of Canada?", "NULL"),
("Who is the Prime Minister of the United Kingdom?", "NULL"),
("What's the deepest part of the ocean?", "NULL"),
("Who composed the Fifth Symphony?", "NULL"),
("What's the largest country in the world?", "NULL"),
("Who is the fastest man in the world?", "NULL"),
("What's the national dish of Spain?", "NULL"),
("Who won the Nobel Prize in Literature last year?", "NULL"),
("What's the smallest planet in the solar system?", "NULL"),
("Who is the current Pope?", "NULL"),
("What's the national anthem of France?", "NULL"),
("Who was the first man on the moon?", "NULL"),
("What's the oldest civilization in the world?", "NULL"),
("Who is the most followed person on Instagram?", "NULL"),
("What's the most spoken language in the world?", "NULL"),
("Who is the director of 'Inception'?", "NULL"),
("What's the national fruit of New Zealand?", "NULL"),
("What's the weather like in London?", "NULL"),
("Can you tell me a fun fact about cats?", "NULL"),
("What's the current time in Tokyo?", "NULL"),
("Can you recommend a good movie?", "NULL"),
("What's the latest sports news?", "NULL"),
("Tell me an interesting historical event.", "NULL"),
("What's your favorite science fact?", "NULL"),
("Can you play a trivia game?", "NULL"),
("What's the capital of Sweden?", "NULL"),
("Who won the last season of 'The Voice'?", "NULL"),
("What's the tallest building in the world?", "NULL"),
("Who is the current Prime Minister of Japan?", "NULL"),
("What's the distance to Mars?", "NULL"),
("Can you set an alarm for me?", "NULL"),
("What's the secret to happiness?", "NULL"),
("Can you tell me a brain teaser?", "NULL"),
("What's the population of Brazil?", "NULL"),
("Who wrote '1984'?", "NULL"),
("What's the longest highway in the world?", "NULL"),
("Can you translate 'good morning' to Italian?", "NULL"),
("What's the speed of sound?", "NULL"),
("Who invented the internet?", "NULL"),
("What's the currency of Switzerland?", "NULL"),
("Who sculpted 'The Thinker'?", "NULL"),
("What's the largest continent in the world?", "NULL"),
("Who is the most successful Olympian?", "NULL"),
("What's the national animal of Scotland?", "NULL"),
("Who discovered penicillin?", "NULL"),
("What's the lifespan of a parrot?", "NULL"),
("Can you tell me a palindrome?", "NULL"),
("What's the national flower of the United States?", "NULL"),
("Who is the author of 'The Hobbit'?", "NULL"),
("What's the diameter of Jupiter?", "NULL"),
("Who was the first woman to win a Nobel Prize?", "NULL"),
("What's the national bird of Australia?", "NULL"),
("Who is the CEO of Google?", "NULL"),
("What's the highest grossing book of all time?", "NULL"),
("Can you tell me a limerick?", "NULL"),
("What's the national sport of Japan?", "NULL"),
("Who is the President of Russia?", "NULL"),
("What's the deepest cave in the world?", "NULL"),
("Who composed the 'Four Seasons'?", "NULL"),
("What's the smallest ocean in the world?", "NULL"),
("Who is the fastest woman in the world?", "NULL"),
("What's the national dish of Germany?", "NULL"),
("Who won the Nobel Prize in Peace last year?", "NULL"),
("What's the hottest planet in the solar system?", "NULL"),
("Who is the current Secretary-General of the United Nations?", "NULL"),
("What's the national anthem of Australia?", "NULL"),
("Who was the first woman in the moon?", "NULL"),
("Who is the author of 'The Catcher in the Rye'?", "NULL"),
]
```
%% Cell type:code id: tags:
``` python
import statistics
def max_threshold_test(threshold: float, scores: list):
return max(scores) > threshold
def mean_threshold_test(threshold: float, scores: list):
return statistics.mean(scores) > threshold
```
%% Cell type:code id: tags:
``` python
from tqdm.auto import tqdm
import time
results = {}
# thresholds = [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95]
thresholds = [0.75, 0.77, 0.79, 0.81, 0.83, 0.85, 0.87, 0.89, 0.91]
threshold_method = 'mean' # 'mean', 'max'
for q, expected in tqdm(queries):
# Attempt Query 3 Times.
out = 'UNDEFINED_CLASS' # Initialize actual_decision here
all_attempts_failed = True # Initialize flag here
for i in range(3):
try:
start_time = time.time() # Start timer
out = dl._query(q, top_k=5)
end_time = time.time() # End timer
all_attempts_failed = False # If we reach this line, the attempt was successful
break
except Exception as e:
print(f"\t\t\tAttempt {i+1} failed with error: {str(e)}")
if i < 2: # Don't sleep after the last attempt
time.sleep(5)
if all_attempts_failed:
print("\t\t\tAll attempts failed. Skipping this utterance.")
continue # Skip to the next utterance
# Determine Top Class and the Cosine-Similarity Scores of Vectors that Contributed to Top Class score.
top_class, top_class_scores = dl._semantic_classify(query_results=out)
# test if the top score is above the threshold for range of thresholds
for threshold in thresholds:
if threshold not in results:
results[threshold] = []
if threshold_method == 'mean':
class_pass = mean_threshold_test(threshold, top_class_scores)
elif threshold_method == 'max':
class_pass = max_threshold_test(threshold, top_class_scores)
if class_pass:
pass
else:
top_class = "NULL"
correct = top_class == expected
results[threshold].append(correct)
```
%% Output
100%|██████████| 252/252 [07:25<00:00, 1.77s/it]
%% Cell type:code id: tags:
``` python
for k, v in results.items():
print(f"Threshold: {k}, Accuracy: {sum(v) / len(v)}")
```
%% Output
Threshold: 0.75, Accuracy: 0.5317460317460317
Threshold: 0.77, Accuracy: 0.6626984126984127
Threshold: 0.79, Accuracy: 0.8015873015873016
Threshold: 0.81, Accuracy: 0.8531746031746031
Threshold: 0.83, Accuracy: 0.7420634920634921
Threshold: 0.85, Accuracy: 0.6388888888888888
Threshold: 0.87, Accuracy: 0.5714285714285714
Threshold: 0.89, Accuracy: 0.5158730158730159
Threshold: 0.91, Accuracy: 0.503968253968254
%% Cell type:code id: tags:
``` python
```
decision_layer/__init__.py deleted 100644 → 0
+ 0
− 1
View file @ b2dd6e39
from decision_layer.decision_layer import DecisionLayer
\ No newline at end of file
decision_layer/encoders/__init__.py deleted 100644 → 0
+ 0
− 4
View file @ b2dd6e39
from decision_layer.encoders.base import BaseEncoder
from decision_layer.encoders.cohere import CohereEncoder
from decision_layer.encoders.huggingface import HuggingFaceEncoder
from decision_layer.encoders.openai import OpenAIEncoder
\ No newline at end of file
pyproject.toml
+ 6
− 3
View file @ eecc8281
[tool.poetry] [tool.poetry]
name = "decision-layer" name = "semantic-router"
version = "0.0.1" version = "0.0.1"
description = "Super fast decision layer for AI" description = "Super fast semantic router for AI decision making"
authors = ["James Briggs <james@aurelio.ai>"] authors = [
"James Briggs <james@aurelio.ai>",
"Siraj Aizlewood <siraj@aurelio.ai>"
]
readme = "README.md" readme = "README.md"
[tool.poetry.dependencies] [tool.poetry.dependencies]
......
results1.csv deleted 100644 → 0
+ 0
− 4027
View file @ b2dd6e39
Source diff could not be displayed: it is too large. Options to address this: view the blob.
semantic_router/__init__.py 0 → 100644
+ 1
− 0
View file @ eecc8281
from semantic_router.layer import DecisionLayer
\ No newline at end of file
semantic_router/encoders/__init__.py 0 → 100644
+ 4
− 0
View file @ eecc8281
from semantic_router.encoders.base import BaseEncoder
from semantic_router.encoders.cohere import CohereEncoder
from semantic_router.encoders.huggingface import HuggingFaceEncoder
from semantic_router.encoders.openai import OpenAIEncoder
\ No newline at end of file
decision_layer/encoders/base.py → semantic_router/encoders/base.py
+ 0
− 0
View file @ eecc8281
File moved
decision_layer/encoders/cohere.py → semantic_router/encoders/cohere.py
+ 3
− 3
View file @ eecc8281
import os import os
import cohere import cohere
from decision_layer.encoders import BaseEncoder from semantic_router.encoders import BaseEncoder
class CohereEncoder(BaseEncoder): class CohereEncoder(BaseEncoder):
client: cohere.Client | None client: cohere.Client | None
def __init__(self, name: str, cohere_api_key: str | None = None): def __init__(self, name: str = "embed-english-v3.0", cohere_api_key: str | None = None):
super().__init__(name=name, client=None) super().__init__(name=name, client=None)
cohere_api_key = cohere_api_key or os.getenv("COHERE_API_KEY") cohere_api_key = cohere_api_key or os.getenv("COHERE_API_KEY")
if cohere_api_key is None: if cohere_api_key is None:
...@@ -17,6 +17,6 @@ class CohereEncoder(BaseEncoder): ...@@ -17,6 +17,6 @@ class CohereEncoder(BaseEncoder):
else: else:
input_type = "search_document" input_type = "search_document"
embeds = self.client.embed( embeds = self.client.embed(
texts, input_type=input_type, model="embed-english-v3.0" texts, input_type=input_type, model=self.name
) )
return embeds.embeddings return embeds.embeddings
\ No newline at end of file
decision_layer/encoders/huggingface.py → semantic_router/encoders/huggingface.py
+ 1
− 1
View file @ eecc8281
from decision_layer.encoders import BaseEncoder from semantic_router.encoders import BaseEncoder
class HuggingFaceEncoder(BaseEncoder): class HuggingFaceEncoder(BaseEncoder):
......
decision_layer/encoders/openai.py → semantic_router/encoders/openai.py
+ 1
− 1
View file @ eecc8281
import os import os
from decision_layer.encoders import BaseEncoder from semantic_router.encoders import BaseEncoder
import openai import openai
from time import time from time import time
......
decision_layer/decision_layer.py → semantic_router/layer.py
+ 23
− 16
View file @ eecc8281
from decision_layer.encoders import BaseEncoder from semantic_router.encoders import BaseEncoder, OpenAIEncoder, CohereEncoder
from decision_layer.schema import Decision from semantic_router.schema import Decision
import numpy as np import numpy as np
from numpy.linalg import norm from numpy.linalg import norm
class DecisionLayer: class DecisionLayer:
index = None index = None
categories = None categories = None
threshold = 0.82
def __init__(self, encoder: BaseEncoder, decisions: list[Decision] = []): def __init__(self, encoder: BaseEncoder, decisions: list[Decision] = []):
self.encoder = encoder self.encoder = encoder
self.embeddings_classified = False # decide on default threshold based on encoder
if isinstance(encoder, OpenAIEncoder):
self.threshold = 0.82
elif isinstance(encoder, CohereEncoder):
self.threshold = 0.3
else:
self.threshold = 0.82
# if decisions list has been passed, we initialize index now # if decisions list has been passed, we initialize index now
if decisions: if decisions:
# initialize index now # initialize index now
for decision in decisions: for decision in decisions:
self._add_decision(decision=decision) self._add_decision(decision=decision)
def __call__(self, text: str) -> str | None:
def __call__(self, text: str, _method: str='raw', _threshold: float=0.5):
results = self._query(text) results = self._query(text)
top_class, top_class_scores = self._semantic_classify(results) top_class, top_class_scores = self._semantic_classify(results)
# TODO: Once we determine a threshold methodlogy, we can use it here. passed = self._pass_threshold(top_class_scores, self.threshold)
if passed:
return top_class
else:
return None
def add(self, decision: Decision): def add(self, decision: Decision):
self._add_decision(devision=decision) self._add_decision(devision=decision)
...@@ -41,12 +53,6 @@ class DecisionLayer: ...@@ -41,12 +53,6 @@ class DecisionLayer:
embed_arr = np.array(embeds) embed_arr = np.array(embeds)
self.index = np.concatenate([self.index, embed_arr]) self.index = np.concatenate([self.index, embed_arr])
def _cosine_similarity(self, v1, v2):
"""Compute the dot product between two embeddings using numpy functions."""
np_v1 = np.array(v1)
np_v2 = np.array(v2)
return np.dot(np_v1, np_v2) / (np.linalg.norm(np_v1) * np.linalg.norm(np_v2))
def _query(self, text: str, top_k: int=5): def _query(self, text: str, top_k: int=5):
"""Given some text, encodes and searches the index vector space to """Given some text, encodes and searches the index vector space to
retrieve the top_k most similar records. retrieve the top_k most similar records.
...@@ -64,10 +70,8 @@ class DecisionLayer: ...@@ -64,10 +70,8 @@ class DecisionLayer:
return [ return [
{"decision": d, "score": s.item()} for d, s in zip(decisions, scores) {"decision": d, "score": s.item()} for d, s in zip(decisions, scores)
] ]
def _semantic_classify(self, query_results: dict): def _semantic_classify(self, query_results: dict):
scores_by_class = {} scores_by_class = {}
for result in query_results: for result in query_results:
score = result['score'] score = result['score']
...@@ -76,9 +80,12 @@ class DecisionLayer: ...@@ -76,9 +80,12 @@ class DecisionLayer:
scores_by_class[decision].append(score) scores_by_class[decision].append(score)
else: else:
scores_by_class[decision] = [score] scores_by_class[decision] = [score]
# Calculate total score for each class # Calculate total score for each class
total_scores = {decision: sum(scores) for decision, scores in scores_by_class.items()} total_scores = {decision: sum(scores) for decision, scores in scores_by_class.items()}
top_class = max(total_scores, key=total_scores.get, default=None) top_class = max(total_scores, key=total_scores.get, default=None)
# Return the top class and its associated scores # Return the top class and its associated scores
return top_class, scores_by_class.get(top_class, []) return str(top_class), scores_by_class.get(top_class, [])
def _pass_threshold(self, scores: list[float], threshold: float):
"""Returns true if the threshold has been passed."""
return max(scores) > threshold
decision_layer/schema.py → semantic_router/schema.py
+ 1
− 1
View file @ eecc8281
from enum import Enum from enum import Enum
from pydantic import BaseModel from pydantic import BaseModel
from pydantic.dataclasses import dataclass from pydantic.dataclasses import dataclass
from decision_layer.encoders import ( from semantic_router.encoders import (
BaseEncoder, BaseEncoder,
HuggingFaceEncoder, HuggingFaceEncoder,
OpenAIEncoder, OpenAIEncoder,
......
01_function_tests.ipynb → walkthrough.ipynb
+ 226
− 0
View file @ eecc8281
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Decision Layer Walkthrough # Semantic Router Walkthrough
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The decision layer library can be used as a super fast decision making layer on top of LLMs. That means that rather than waiting on a slow agent to decide what to do, we can use the magic of semantic vector space to make decisions. Cutting decision making time down from seconds to milliseconds. The Semantic Router library can be used as a super fast decision making layer on top of LLMs. That means rather than waiting on a slow agent to decide what to do, we can use the magic of semantic vector space to make decisions. Cutting decision making time down from seconds to milliseconds.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Getting Started ## Getting Started
%% Cell type:markdown id: tags:
We start by installing the library:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
!pip install -qU \ !pip install -qU semantic-router==0.0.1
decision-layer
``` ```
%% Output
[notice] A new release of pip is available: 23.1.2 -> 23.3.1
[notice] To update, run: python.exe -m pip install --upgrade pip
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
We start by defining a dictionary mapping decisions to example phrases that should trigger those decisions. We start by defining a dictionary mapping decisions to example phrases that should trigger those decisions.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
from decision_layer.schema import Decision from semantic_router.schema import Decision
politics = Decision( politics = Decision(
name="politics", name="politics",
utterances=[ utterances=[
"isn't politics the best thing ever", "isn't politics the best thing ever",
"why don't you tell me about your political opinions", "why don't you tell me about your political opinions",
"don't you just love the president" "don't you just love the president"
"don't you just hate the president", "don't you just hate the president",
"they're going to destroy this country!", "they're going to destroy this country!",
"they will save the country!", "they will save the country!"
"did you hear about the new goverment proposal regarding the ownership of cats and dogs",
] ]
) )
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Let's define another for good measure: Let's define another for good measure:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
chitchat = Decision( chitchat = Decision(
name="chitchat", name="chitchat",
utterances=[ utterances=[
"how's the weather today?", "how's the weather today?",
"how are things going?", "how are things going?",
"lovely weather today", "lovely weather today",
"the weather is horrendous", "the weather is horrendous",
"let's go to the chippy", "let's go to the chippy"
"it's raining cats and dogs",
] ]
) )
decisions = [politics, chitchat] decisions = [politics, chitchat]
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Now we initialize our embedding model (we will add support for Hugging Face): Now we initialize our embedding model:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
from decision_layer.encoders import OpenAIEncoder from semantic_router.encoders import CohereEncoder
from getpass import getpass
import os import os
encoder = OpenAIEncoder(name="text-embedding-ada-002") os.environ["COHERE_API_KEY"] = getpass("Enter Cohere API Key: ")
encoder = CohereEncoder()
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Now we define the `DecisionLayer`. When called, the decision layer will consume text (a query) and output the category (`Decision`) it belongs to — for now we can only `_query` and get the most similar `Decision` `utterances`. Now we define the `DecisionLayer`. When called, the decision layer will consume text (a query) and output the category (`Decision`) it belongs to — to initialize a `DecisionLayer` we need our `encoder` model and a list of `decisions`.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
from decision_layer import DecisionLayer from semantic_router import DecisionLayer
dl = DecisionLayer(encoder=encoder, decisions=decisions) dl = DecisionLayer(encoder=encoder, decisions=decisions)
``` ```
%% Cell type:code id: tags:
``` python
out = dl._query("don't you love politics?")
out
```
%% Output
[{'decision': 'politics', 'score': 0.22792677421560453},
{'decision': 'politics', 'score': 0.2315237823644528},
{'decision': 'politics', 'score': 0.2516642096551168},
{'decision': 'politics', 'score': 0.2531645714220874},
{'decision': 'politics', 'score': 0.2566108224655662}]
%% Cell type:markdown id: tags:
Using the most similar `Decision` `utterances` and their `cosine similarity scores`, use `simple_classify` to apply scoring a secondary scoring system which chooses the `decision` that the utterance belongs to.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Here we use `apply_tan=True`, which means that a `tan` function is assigned to each score boosting the score of `decisions` whose datapoints had greater `cosine similarlity` and reducing the score of those which had lower `cosine similarity`. Now we can test it:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
decision, scores_by_category = dl.simple_classify(query_results=out, apply_tan=True) dl("don't you love politics?")
decision
``` ```
%% Output %% Output
'politics' 'politics'
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
scores_by_category dl("how's the weather today?")
``` ```
%% Output %% Output
{'politics': 2.018519173992354} 'chitchat'
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The correct category was chosen. Let's try again for a less clear-cut case: Both are classified accurately, what if we send a query that is unrelated to our existing `Decision` objects?
%% Cell type:code id: tags:
``` python
out = dl._query("i love cats and dogs!")
out
```
%% Output
[{'decision': 'chitchat', 'score': 0.22320888353212376},
{'decision': 'politics', 'score': 0.22367029584935166},
{'decision': 'politics', 'score': 0.2274250403127478},
{'decision': 'politics', 'score': 0.23451692377042876},
{'decision': 'chitchat', 'score': 0.24924083653953585}]
%% Cell type:code id: tags:
``` python
decision, scores_by_category = dl.simple_classify(query_results=out, apply_tan=True)
decision
```
%% Output
'politics'
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
scores_by_category dl("I'm interested in learning about llama 2")
``` ```
%% Output %% Cell type:markdown id: tags:
{'chitchat': 0.7785435459589187, 'politics': 1.1258003022715952}
%% Cell type:code id: tags:
``` python
dl.categories
```
%% Output
array(['politics', 'politics', 'politics', 'politics', 'politics', In this case, we return `None` because no matches were identified.
'politics', 'chitchat', 'chitchat', 'chitchat', 'chitchat',
'chitchat', 'chitchat'], dtype='<U8')
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

:feet::heart: Strive to be the person your dogs believe you are :heart::feet: