LLM 前沿技术参考资料

核心架构演进

Transformer 变体

• 原始 Transformer: "Attention Is All You Need" (Vaswani et al., 2017)
• GPT 系列: GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023)
• LLaMA 系列: LLaMA (Touvron et al., 2023), LLaMA 2/3 (Meta, 2023-2024)
• Mistral/Mixtral: Mistral 7B, Mixtral 8x7B MoE (Mistral AI, 2023-2024)
• Qwen 系列: Qwen2, Qwen2.5 (Alibaba, 2024)
• DeepSeek 系列: DeepSeek-V2 MoE, DeepSeek-V3 (DeepSeek, 2024-2025)

MoE (Mixture of Experts)

• 开创论文: "Outrageously Large Neural Networks" (Shazeer et al., 2017)
• GShard: Lepikhin et al., 2020
• Switch Transformer: Fedus et al., 2022
• DeepSeek-V2 MoE: 细粒度专家 + 共享专家设计
• Mixtral: Top-2 路由, 8 专家 MoE

长上下文技术

• RoPE 扩展: "RoFormer" (Su et al., 2021)
• YaRN: "Yet another RoPE extensioN method" (Peng et al., 2023)
• ALiBi: "Train Short, Test Long" (Press et al., 2022)
• Ring Attention: Liu et al., 2023
• FlashAttention: Dao et al., 2022; FlashAttention-2, Dao, 2023

推测解码

• Speculative Decoding: Leviathan et al., 2023; Chen et al., 2023
• Medusa: Cai et al., 2024 — 多头并行解码
• Eagle: Li et al., 2024 — 特征层推测解码

训练技术

• RLHF: "Training language models to follow instructions" (Ouyang et al., 2022)
• DPO: "Direct Preference Optimization" (Rafailov et al., 2023)
• ORPO: "ORPO: Monolithic Preference Optimization" (Hong et al., 2024)
• KTO: "Kahneman-Tversky Optimization" (Ethayarajh et al., 2024)
• GRPO: DeepSeek-V2 技术报告中的组相对策略优化

关键项目

• Hugging Face Transformers: https://github.com/huggingface/transformers
• vLLM: https://github.com/vllm-project/vllm
• Axolotl: https://github.com/OpenAccess-AI-Collective/axolotl
• LLaMA-Factory: https://github.com/hiyouga/LLaMA-Factory

评估基准

• MMLU: Massive Multitask Language Understanding
• HumanEval: Code generation
• GSM8K: Math reasoning
• IFEval: Instruction following
• Arena Hard: Chatbot arena evaluation