There is no single official “throughput number” for DeepSeek-V3.2 on A100 GPUs, and DeepSeek’s own technical report focuses more on cost per million tokens and asymptotic complexity than on tokens-per-second on specific NVIDIA SKUs. The published evaluations for V3.2-Exp benchmark inference costs on H800 clusters and show that, for long contexts, DSA substantially cuts cost per million tokens compared to V3.1-Terminus, both for prefilling and decoding.:contentReference[oaicite:13]{index=13} However, those numbers are cluster- and engine-specific, and H800 is not identical to A100. As of now, open documentation and vLLM/SGLang recipes explain how to run the model on NVIDIA GPUs but do not provide a canonical “X tokens/s on A100 80GB” guarantee.:contentReference[oaicite:14]{index=14} So any throughput figures you see in blogs or benchmarks should be treated as environment-specific, not as specs of the model itself.
Practically, you should think in terms of factors rather than a single number. Throughput for a 37B-active MoE model like V3.2-Exp depends heavily on (1) context length, (2) batch size and number of concurrent sequences, (3) quantization (FP8 vs BF16), (4) tensor/data parallelism configuration, and (5) the inference engine (vLLM, SGLang, TGI, etc.). On A100, one complication is that native FP8 isn’t supported, so some pipelines convert part of the graph to BF16, which increases memory consumption and can reduce effective throughput or even cause out-of-memory unless you shard the model across many GPUs. The Hugging Face discussions around V3.x and V3.2-Exp note that larger parallelism degrees (e.g., more than 16-way model parallel) can be needed to run BF16 configs without OOM, which in turn influences how much batch you can push per GPU.:contentReference[oaicite:15]{index=15} Engines like vLLM and SGLang add their own optimizations—PagedAttention, DSA-aware kernels, data-parallel attention—that can heavily tilt the trade-off between latency and total tokens/sec.:contentReference[oaicite:16]{index=16}
For a production system, the only numbers that really matter are the ones on your stack, so the recommended approach is to benchmark DeepSeek-V3.2 with the same engine, batch shapes, and context patterns you expect in real traffic. Start with a simple latency/throughput matrix: vary (a) context length (e.g., 2k / 8k / 32k), (b) batch size, and © number of GPUs, and record both tokens/sec and cost per million tokens relative to your GPU hourly price. The technical report’s cost-per-million-tokens curves give you a useful sanity check that you are seeing similar relative gains vs V3.1 for long contexts.:contentReference[oaicite:17]{index=17} In RAG or analytic workloads that rely on a vector database such as Milvus or Zilliz Cloud, you can boost effective throughput more by shortening LLM context—storing long-term knowledge in the vector store and retrieving only top-k results—than by micro-optimizing GPU kernels. For A100 in particular, that combination—moderate context via retrieval + careful sharding and batching—will usually yield better tokens-per-dollar than chasing an absolute tokens-per-second headline figure.