I

inference

🔧🔗https://github.com/vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

🔧🔗https://github.com/sgl-project/sglang

SGLang is a fast serving framework for large language models and vision language models.

🔧🔗https://github.com/vllm-project/vllm-ascend Community maintained hardware plugin for vLLM on Ascend

https://github.com/roboflow/inference A fast, easy-to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models.

🔧🔗https://github.com/SYSTRAN/faster-whisper

Faster Whisper transcription with CTranslate2

🔧🔗https://github.com/nbursa/inception-core-server

A modular, extensible Rust-based server providing short-term, long-term, and latent memory services, a chat endpoint with a BaseAgent + Sentience DSL, and integration with ChromaDB and LLM services.

🔧🔗https://github.com/flashinfer-ai/debug-print

Debug print operator for cudagraph debugging