Skip to content

GitLab

Explore

Sign in

Primary navigation

Project

LlamaIndexTS
- Activity
- Members
- Labels
- Issues
- Issue boards
- Milestones
- Iterations
- Wiki
- Environments
- Terraform modules
- Incidents
- Service Desk

Snippets Groups Projects

18ec1f2f

Unverified Commit 18ec1f2f authored 4 months ago by Alex Yang Committed by GitHub 4 months ago

Downloads
- Patches
- Plain Diff

chore: separate tokenizers (#1454)

parent b0fbd8b5

No related branches found

No related tags found

No related merge requests found

Changes 28

Hide whitespace changes

Inline Side-by-side

Showing

packages/env/src/internal/tokenizers/types.ts 0 additions, 0 deletions

packages/env/src/internal/tokenizers/types.ts
packages/env/src/tokenizers.browser.ts 5 additions, 0 deletions

packages/env/src/tokenizers.browser.ts
packages/env/src/tokenizers.edge-light.ts 5 additions, 0 deletions

packages/env/src/tokenizers.edge-light.ts
packages/env/src/tokenizers.ts 5 additions, 0 deletions

packages/env/src/tokenizers.ts
packages/env/src/tokenizers.workerd.ts 5 additions, 0 deletions

packages/env/src/tokenizers.workerd.ts
packages/env/tests/tokenizer.test.ts 13 additions, 3 deletions

packages/env/tests/tokenizer.test.ts
packages/providers/openai/src/embedding.ts 2 additions, 1 deletion

packages/providers/openai/src/embedding.ts
packages/providers/openai/src/llm.ts 2 additions, 1 deletion

packages/providers/openai/src/llm.ts

with 37 additions and 5 deletions

Loading

0% Loading or .

You are about to add 0 people to the discussion. Proceed with caution.

Finish editing this message first!

Please register or sign in to comment

Strive to be the person your dogs believe you are