RepositoryPython
DE

deepseek-ai/DeepSeek-V3

No description provided for this repository.

104.9kstars
16.7kforks
252issues
10views

🚀 DeepSeek-V3: A 671B MoE Model Built for Efficiency and Scale

DeepSeek-V3 delivers state-of-the-art open-source performance with only 2.788M H800 GPU hours of training.

DeepSeek-V3 is a powerful Mixture-of-Experts (MoE) language model developed by DeepSeek AI. With 671B total parameters and 37B activated per token, it combines cutting-edge architecture techniques like Multi-head Latent Attention (MLA) and auxiliary-loss-free load balancing to achieve both high performance and training stability. Pre-trained on 14.8 trillion tokens, it supports long contexts up to 128K tokens and rivals top closed-source models.

✨ Highlights

  • Mixture-of-Experts (MoE): 671B total parameters with only 37B activated per token for efficient inference.
  • Multi-head Latent Attention (MLA): Reduces memory usage and accelerates attention computation.
  • Auxiliary-Loss-Free Load Balancing: Improves expert utilization without hurting model performance.
  • Multi-Token Prediction (MTP): Enhances model accuracy and enables speculative decoding for faster inference.
  • FP8 Mixed Precision Training: First large-scale validation of FP8 for cost-effective and stable training.
  • Co-Designed Communication Optimization: Overcomes cross-node MoE bottlenecks for near-full compute-communication overlap.
  • Knowledge Distillation from DeepSeek-R1: Boosts reasoning capabilities using long-chain-of-thought supervision.
  • Open Source & Accessible: Available on Hugging Face with MIT code license and model agreement.

💡 Why it matters

DeepSeek-V3 stands out by delivering exceptional performance while maintaining training efficiency and stability. Unlike many large models that require extensive compute and suffer from instability, DeepSeek-V3 trains smoothly without rollbacks and achieves results comparable to leading proprietary models. This makes it a practical choice for researchers and developers looking to build advanced LLM-powered applications with an open and scalable foundation.


#DeepSeekAI #OpenSourceLLM #MixtureOfExperts #TiniX