推理引擎参考资料

vLLM

• GitHub: https://github.com/vllm-project/vllm
• 核心论文: "Efficient Memory Management for Large Language Model Serving with PagedAttention" (Kwon et al., 2023)
• 关键特性:
  • PagedAttention: 虚拟内存管理 KV Cache
  • Continuous Batching: 迭代级调度
  • Prefix Caching: 共享前缀复用
  • Speculative Decoding: 小模型辅助推测
  • LoRA Serving: 多 LoRA 动态切换
  • Chunked Prefill: 分块预填充

SGLang

• GitHub: https://github.com/sgl-project/sglang
• 核心论文: "SGLang: Efficient Execution of Structured Language Model Programs" (Zheng et al., 2024)
• 关键特性:
  • RadixAttention: 基数树前缀共享
  • Continuation-based API: 原生多轮对话
  • Runtime-based batching: 自动请求调度
  • Grammar-constrained decoding: 结构化输出
  • Overlap scheduling: 计算通信重叠

TensorRT-LLM

• GitHub: https://github.com/NVIDIA/TensorRT-LLM
• 核心特性:
  • NVIDIA GPU 专用优化
  • Kernel fusion: 算子融合
  • Weight-only quantization: INT4/INT8 权重量化
  • KV Cache 优化: 连续内存布局
  • In-flight batching: 飞行中批处理
  • Multi-query/grouped-query attention 优化

LMDeploy

• GitHub: https://github.com/InternLM/lmdeploy
• 核心特性: Turbomind 推理引擎, W4A16 量化, KV Cache 量化

量化技术

• GPTQ: "Accurate Quantization for Generative Pre-trained Transformers" (Frantar et al., 2023)
• AWQ: "Activation-aware Weight Quantization" (Lin et al., 2024)
• SmoothQuant: "Accurate and Efficient Post-Training Quantization" (Xiao et al., 2023)
• GGUF/GGML: llama.cpp 量化格式
• FP8: NVIDIA H100 原生 FP8 支持
• Quantization-Aware Training (QAT): 训练中量化感知

解码策略

• Greedy Decoding: 贪心解码
• Beam Search: 束搜索
• Top-k Sampling: 限制候选数
• Top-p (Nucleus) Sampling: 概率质量截断
• Typical Sampling: 信息论采样
• Min-p Sampling: 最小概率阈值
• Repetition Penalty: 重复惩罚
• Grammar-constrained Decoding: 语法约束解码 (JSON Schema)

关键优化

• KV Cache: 键值缓存复用
• Continuous Batching: 迭代级批处理
• Prefix Caching/RadixAttention: 前缀共享
• Speculative Decoding: 推测加速
• Chunked Prefill: 分块预填充减少延迟
• Tensor Parallelism: 张量并行
• Pipeline Parallelism: 流水线并行