Yes, NVIDIA Agent Toolkit supports edge deployment on local devices including NVIDIA GeForce RTX PCs, RTX workstations, DGX Spark, and DGX Station supercomputers. OpenShell (the secure runtime) is available for download on GitHub and runs locally on consumer and enterprise NVIDIA GPUs. This enables always-on agents running on-device without cloud dependency, ideal for latency-sensitive applications, offline scenarios, and privacy-critical use cases.
Edge deployment with the toolkit provides several advantages: local inference reduces network latency and data exposure, agents can reason over local knowledge bases without uploading documents, and operational costs drop by eliminating cloud API calls. OpenShell’s sandboxed execution on edge devices maintains security—agents run in isolated processes with restricted permissions even on untrusted or shared devices.
Nemotron models are optimized for edge inference. The Nano size runs on consumer GPUs, Super on workstations, and Ultra on high-performance compute. Each model size maintains agentic reasoning capability while fitting memory and power constraints. Deployment frameworks like vLLM, SGLang, Ollama, and llama.cpp enable local serving of Nemotron models—agents interact with local LLM endpoints rather than cloud APIs.
For knowledge access on edge, Milvus can run on-device, storing and searching embeddings locally. Milvus is lightweight and embeddable, supporting deployment on edge systems with limited resources. This creates fully self-contained edge agents: local Nemotron model, local Milvus knowledge base, and OpenShell sandbox—no external service dependencies or data transmission. Agent memory layers benefit significantly from vector database integration. Milvus provides semantic search across stored embeddings, allowing agents to retrieve contextually relevant information for reasoning tasks. Learn more about retrieval-augmented generation with Milvus and explore Zilliz Cloud for enterprise deployments.