Running DeepSeek-V3.2-Exp efficiently on-prem or self-hosted requires serious hardware, because you’re dealing with a 685B-parameter MoE model, even though only ~37B parameters are active per token. DeepSeek and vLLM’s recipes focus on multi-GPU setups, with recent guides optimized for Nvidia Blackwell GPUs (GB200/B200) and also strong support via SGLang for Nvidia and several non-Nvidia accelerators. As a rough rule of thumb, you should expect to use many high-memory GPUs (for example, 8–16× 80GB-class GPUs or equivalent) if you want decent batch sizes, long context (e.g., 128k), and good latency. Exact sizing depends on your tensor-parallel configuration, quantization strategy, and target context length.
Independent hardware requirement guides for DeepSeek models recommend matching GPU count and VRAM to both model size and context length, and they stress that MoE models like DeepSeek V3.x are bandwidth-sensitive: you need enough interconnect speed (NVLink/NVSwitch or equivalent) to avoid communication becoming the bottleneck. If you’re experimenting or doing low-throughput workloads, you might squeeze by with fewer GPUs and lower context limits, but you’ll quickly run into memory or performance ceilings if you try to push large batches or very long prompts. For many teams, the practical answer is to start on a managed DeepSeek API or a hosted platform that abstracts the cluster and lets you focus on latency, tokens, and cost rather than sharding schemes.
If your main use case is RAG over a vector database such as Milvus or Zilliz Cloud, you can often scale down your LLM hardware by shifting more work into the retrieval layer. Instead of leaning on 128k-token prompts, keep prompts in a more modest range (8–32k), store most of your corpus in Milvus/Zilliz Cloud, and tune your retrieval so that you only pass narrow, high-signal context into V3.2. That lets you run smaller batches and shorter prompts, which in turn makes it easier to run V3.2 efficiently on fewer GPUs or slightly older cards. Put differently: hardware matters a lot, but system design (RAG vs pure long context, batching strategy, quantization) is usually the bigger multiplier on how “efficient” your deployment feels day-to-day.