Skip to content

GitLab

Explore

Sign in

Primary navigation

Project

SwissArmyTransformer
- Activity
- Members
- Labels
- Issues
- Issue boards
- Milestones
- Iterations
- Wiki
- Environments
- Terraform modules
- Incidents
- Service Desk

Snippets Groups Projects

229f37e1

Commit 229f37e1 authored 3 years ago by Ming Ding

Downloads
- Patches
- Plain Diff

tokenization

parent 142df57c

No related branches found

No related tags found

Loading

Changes 6

Hide whitespace changes

Inline Side-by-side

Showing

tokenization/__init__.py 36 additions, 0 deletions

tokenization/__init__.py
tokenization/cogview/__init__.py 16 additions, 0 deletions

tokenization/cogview/__init__.py
tokenization/cogview/sp_tokenizer.py 150 additions, 0 deletions

tokenization/cogview/sp_tokenizer.py
tokenization/cogview/templates.py 77 additions, 0 deletions

tokenization/cogview/templates.py
tokenization/cogview/unified_tokenizer.py 193 additions, 0 deletions

tokenization/cogview/unified_tokenizer.py
tokenization/cogview/vqvae_tokenizer.py 86 additions, 0 deletions

tokenization/cogview/vqvae_tokenizer.py

with 558 additions and 0 deletions

Loading

0% Loading or .

You are about to add 0 people to the discussion. Proceed with caution.

Finish editing this message first!

Please register or sign in to comment

Strive to be the person your dogs believe you are