Skip to content

GitLab

Explore

Sign in

Primary navigation

Project

SwissArmyTransformer
- Activity
- Members
- Labels
- Issues
- Issue boards
- Milestones
- Iterations
- Wiki
- Environments
- Terraform modules
- Incidents
- Service Desk

Snippets Groups Projects

52b96a84

Unverified Commit 52b96a84 authored 3 years ago by duzx16 Committed by GitHub 3 years ago

Downloads
- Plain Diff

Merge pull request #3 from THUDM/tokenizer

Refactor tokenization

parents 028cfe61 cb9ad4ce

No related branches found

No related tags found

No related merge requests found

Changes 4

Expand all Hide whitespace changes

Inline Side-by-side

Showing

SwissArmyTransformer/tokenization/__init__.py 3 additions, 1 deletion

SwissArmyTransformer/tokenization/__init__.py
SwissArmyTransformer/tokenization/glm/sp_tokenizer.py 65 additions, 117 deletions

SwissArmyTransformer/tokenization/glm/sp_tokenizer.py
SwissArmyTransformer/tokenization/glm/tokenization.py 136 additions, 756 deletions

SwissArmyTransformer/tokenization/glm/tokenization.py
SwissArmyTransformer/tokenization/glm/tokenization_gpt2.py 9 additions, 1 deletion

SwissArmyTransformer/tokenization/glm/tokenization_gpt2.py

with 213 additions and 875 deletions

Loading

0% Loading or .

You are about to add 0 people to the discussion. Proceed with caution.

Finish editing this message first!

Please register or sign in to comment

Strive to be the person your dogs believe you are