L

llm-inference

🔧🔗https://github.com/vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

🔧🔗https://github.com/flashinfer-ai/flashinfer

FlashInfer: Kernel Library for LLM Serving

🕸️🔗flashinfer.ai

🔧🔗https://github.com/google/gemma.cpp

lightweight, standalone C++ inference engine for Google's Gemma models.

https://github.com/InternLM/lmdeploy LMDeploy is a toolkit for compressing, deploying, and serving LLMs. 🔗 lmdeploy.readthedocs.io/en/latest/

🔧🔗https://github.com/modelscope/dash-infer

DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including x86 and ARMv9.

🔧🔗https://github.com/bytedance/ShadowKV ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

https://github.com/coqui-ai/inference-engine Coqui Inference Engine