多模态 AI 参考资料料

视觉模型

Vision Transformer (ViT)

• 论文: "An Image is Worth 16x16 Words" (Dosovitskiy et al., 2021)
• 核心: 图像分块 → 线性嵌入 → Transformer 编码器
• 变体: DeiT, Swin Transformer, CvT, ViT-G

Swin Transformer

• 论文: "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (Liu et al., 2021)
• 核心: 层级结构 + 移位窗口注意力 + 线性复杂度

ConvNeXt

• 论文: "A ConvNet for the 2020s" (Liu et al., 2022)
• 核心: 纯卷积架构, 匹配 ViT 性能

多模态模型

CLIP

• 论文: "Learning Transferable Visual Models From Natural Language Supervision" (Radford et al., 2021)
• 核心: 对比学习, 图文对齐, 零样本分类
• 开源: OpenCLIP — https://github.com/mlfoundations/open_clip

DALL-E 系列

• DALL-E: "Zero-Shot Text-to-Image Generation" (Ramesh et al., 2021)
• DALL-E 2: "Hierarchical Text-Conditional Image Generation" (Ramesh et al., 2022)
• DALL-E 3: 改进对齐与细节

Stable Diffusion

• 论文: "High-Resolution Image Synthesis with Latent Diffusion Models" (Rombach et al., 2022)
• GitHub: https://github.com/CompVis/stable-diffusion
• 变体: SDXL, SD3, Stable Video Diffusion
• 生态: ControlNet, LoRA, IP-Adapter, ComfyUI

LLaVA

• 论文: "Visual Instruction Tuning" (Liu et al., 2024)
• GitHub: https://github.com/haotian-liu/LLaVA
• 核心: CLIP ViT + LLM, 视觉指令微调
• 变体: LLaVA-1.5, LLaVA-NeXT, LLaVA-OneVision

InternVL

• GitHub: https://github.com/OpenGVLab/InternVL
• 核心: InternViT + InternLM, 强视觉理解

Qwen-VL

• GitHub: https://github.com/QwenLM/Qwen-VL
• 核心: Qwen LLM + 视觉编码器, 多语言多模态

CogVLM/CogVL2

• GitHub: https://github.com/THUDM/CogVLM
• 核心: 视觉专家模块, 深度融合

图像分割

SAM (Segment Anything)

• 论文: "Segment Anything" (Kirillov et al., 2023)
• GitHub: https://github.com/facebookresearch/segment-anything
• 核心: Promptable 分割, 零样本泛化

SAM 2

• 论文: "SAM 2: Segment Anything in Images and Videos" (Ravi et al., 2024)
• 核心: 视频分割, 记忆注意力

视频生成

• Sora: OpenAI 视频生成模型
• Stable Video Diffusion: Stability AI 视频模型
• CogVideoX: 清华视频生成
• HunyuanVideo: 腾讯混元视频

3D 生成

• Point-E: OpenAI 3D 点云生成
• Shap-E: OpenAI 3D 隐式表示
• Gaussian Splatting: 3DGS 实时渲染

语音多模态

• Whisper: OpenAI 语音识别
• Bark: Suno 语音生成
• VALL-E: Microsoft 语音合成
• GPT-4o: 原生语音交互

关键论文

• "Flamingo: a Visual Language Model for Few-Shot Learning" (Alayrac et al., 2022)
• "BLIP-2: Bootstrapping Language-Image Pre-training" (Li et al., 2023)
• "Gemini: A Family of Highly Capable Multimodal Models" (Google, 2024)