Skip to content
Snippets Groups Projects
Unverified Commit d1ca16f7 authored by Timothy Carambat's avatar Timothy Carambat Committed by GitHub
Browse files

Add tokenizer improvments via Singleton class and estimation (#3072) 

* Add tokenizer improvments via Singleton class
linting

* dev build

* Estimation fallback when string exceeds a fixed byte size

* Add notice to tiktoken on backend
parent e1af72da
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 125 additions and 29 deletions
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

:feet::heart: Strive to be the person your dogs believe you are :heart::feet: