M

multimodal

🔧🔗https://github.com/modelscope/ms-swift

SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning) Use PEFT or Full-parameter to finetune 400+ LLMs or 100+ MLLMs.

🔧🔗https://github.com/modelscope/modelscope-agent

ModelScope-Agent: An agent framework connecting models in ModelScope with the world

🔧🔗https://github.com/om-ai-lab/VLM-R1 Solve Visual Understanding with Reinforced VLMs

https://github.com/InternLM/HuixiangDou HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance

https://github.com/Stability-AI/stability-sdk SDK for interacting with stability.ai APIs (e.g. stable diffusion inference)

🔧🔗https://github.com/lucidrains/transfusion-pytorch

Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI

https://github.com/InternLM/InternLM-XComposer InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output

https://github.com/princeton-nlp/CharXiv CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

🔧🔗https://github.com/joanrod/star-vector StarVector is a foundation model for SVG generation that transforms vectorization into a code generation task. Using a vision-language modeling architecture, StarVector processes both visual and textual inputs to produce high-quality SVG code with remarkable precision.

🔧🔗https://github.com/om-ai-lab/OmAgent Build multimodal language agents for fast prototype and production

🔧🔗https://github.com/om-ai-lab/VL-CheckList Evaluating Vision & Language Pretraining Models with Objects, Attributes and Relations. [EMNLP 2022]

🔧🔗https://github.com/om-ai-lab/GroundVLP GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection (AAAI 2024)

🔧🔗https://github.com/deepseek-ai/Janus

Janus-Series: Unified Multimodal Understanding and Generation Models

🔧🔗https://github.com/FoundationVision/Groma

[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization

🕸️🔗groma-mllm.github.io/

🔧🔗https://github.com/forhaoliu/language-quantized-autoencoders

Language Quantized AutoEncoders This is a Jax implementation of our work Language Quantized AutoEncoders.