L

llm-inference

🔧🔗https://github.com/vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

🔧🔗https://github.com/flashinfer-ai/flashinfer

FlashInfer: Kernel Library for LLM Serving

🕸️🔗flashinfer.ai

🔧🔗https://github.com/google/gemma.cpp

lightweight, standalone C++ inference engine for Google's Gemma models.

https://github.com/Lightning-AI/litgpt 20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale. 🔗 https://lightning.ai/

https://github.com/InternLM/lmdeploy LMDeploy is a toolkit for compressing, deploying, and serving LLMs. 🔗 lmdeploy.readthedocs.io/en/latest/

https://github.com/mistralai/mistral-inference Official inference library for Mistral models

mistral.ai/

🔧🔗https://github.com/cognitivecomputations/dolphin-logger

This proxy allows you to record your chats to create datasets based on your conversations with any Chat LLM. It supports multiple LLM backends including OpenAI, Anthropic, Google and Ollama.

🔧🔗https://github.com/modelscope/dash-infer

DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including x86 and ARMv9.

🔧🔗https://github.com/nbursa/inception-core-server

A modular, extensible Rust-based server providing short-term, long-term, and latent memory services, a chat endpoint with a BaseAgent + Sentience DSL, and integration with ChromaDB and LLM services.

🔧🔗https://github.com/harishsg993010/neural-state-manipulator A tool for manipulating the internal neural activations of language models

🔧🔗https://github.com/bytedance/ShadowKV ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

https://github.com/huggingface/parler-tts Inference and training library for high-quality TTS models.

The Arcee client for executing domain-adpated language model routines

https://github.com/coqui-ai/inference-engine Coqui Inference Engine

AI stack for interacting with LLMs, Stable Diffusion, Whisper, xTTS and many other AI models https://github.com/noco-ai/spellbook-docker/wiki