L

Large Language Models

🔧🔗https://github.com/vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

🔧🔗https://github.com/flashinfer-ai/flashinfer

FlashInfer: Kernel Library for LLM Serving

🕸️🔗flashinfer.ai

🔧🔗https://github.com/vllm-project/vllm-ascend Community maintained hardware plugin for vLLM on Ascend

🔧🔗https://github.com/EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.

https://github.com/InternLM/lmdeploy LMDeploy is a toolkit for compressing, deploying, and serving LLMs. 🔗 lmdeploy.readthedocs.io/en/latest/

https://github.com/mlc-ai/mlc-llm Universal LLM Deployment Engine with ML Compilation

https://github.com/princeton-nlp/SWE-agent SWE-agent takes a GitHub issue and tries to automatically fix it, using GPT-4, or your LM of choice. It solves 12.47% of bugs in the SWE-bench evaluation set and takes just 1 minute to run.

🔧🔗https://github.com/sgl-project/sglang

SGLang is a fast serving framework for large language models and vision language models.

LLM training in simple, raw C/CUDA LLMs in simple, pure C/CUDA with no need for 245MB of PyTorch or 107MB of cPython. Current focus is on pretraining, in particular reproducing the GPT-2 and GPT-3 miniseries, along with a parallel PyTorch ref

🔧🔗https://github.com/andrewkchan/yalm

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

🔧🔗https://github.com/microsoft/BitBLAS

BitBLAS is a library to support mixed-precision matrix multiplications, especially for quantized LLM deployment.

🔧🔗https://github.com/modelscope/dash-infer

DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including x86 and ARMv9.

https://github.com/janhq/nitro.git now: https://github.com/janhq/cortex.git Drop-in, local AI alternative to the OpenAI stack. Multi-engine (llama.cpp, TensorRT-LLM, ONNX). Powers 👋 Jan

https://github.com/janhq/cortex.tensorrt-llm Cortex.Tensorrt-LLM is a C++ inference library that can be loaded by any server at runtime. It submodules NVIDIA’s TensorRT-LLM for GPU accelerated inference on NVIDIA's GPUs.

🔧🔗https://github.com/QwenLM/qwen.cpp C++ implementation of Qwen-LM

🔧🔗https://github.com/FoundationVision/Groma

[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization

🕸️🔗groma-mllm.github.io/

🔧🔗https://github.com/IST-DASLab/marlin FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

https://github.com/THUDM/APAR APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding

https://github.com/LibreTranslate/RemoveDup Remove duplicates from parallel corpora