M

model-serving

🔧🔗https://github.com/vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

🔧🔗https://github.com/vllm-project/vllm-ascend Community maintained hardware plugin for vLLM on Ascend

https://github.com/Lightning-AI/LitServe Lightning-fast serving engine for AI models. Flexible. Easy. Enterprise-scale. 🔗 https://lightning.ai/docs/litserve

https://github.com/lm-sys/FastChat An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

Resources