tokenization

🔧🔗https://github.com/cohere-ai/magikarp

This repository contains the code and extended results for the paper Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models https://arxiv.org/abs/2405.05417

🔧🔗https://github.com/FoundationVision/OmniTokenizer OmniTokenizer: one model and one weight for image-video joint tokenization.