multimodal

🔧🔗https://github.com/modelscope/ms-swift

SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning) Use PEFT or Full-parameter to finetune 400+ LLMs or 100+ MLLMs.

🔧🔗https://github.com/lucidrains/transfusion-pytorch

Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI

https://github.com/Stability-AI/stability-sdk SDK for interacting with stability.ai APIs (e.g. stable diffusion inference)

https://github.com/InternLM/HuixiangDou HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance

https://github.com/InternLM/InternLM-XComposer InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output

🔧🔗https://github.com/modelscope/modelscope-agent

ModelScope-Agent: An agent framework connecting models in ModelScope with the world

https://github.com/princeton-nlp/CharXiv CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

🔧🔗https://github.com/FoundationVision/Groma

[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization

🕸️🔗groma-mllm.github.io/

🔧🔗https://github.com/forhaoliu/language-quantized-autoencoders

Language Quantized AutoEncoders This is a Jax implementation of our work Language Quantized AutoEncoders.