You should expect voyage-large-2 latency to be workload-dependent, not a single fixed number, because the end-to-end time is the sum of (1) embedding API latency, (2) network round-trip time, and (3) vector search latency in your database. voyage-large-2 is a larger model that outputs 1536-d vectors and supports up to 16K tokens of input, so both the size of your input text and the downstream vector search settings can influence tail latency. Practically: short queries embedded online are usually fast enough for interactive search, while embedding long passages (or embedding many chunks) is better treated as an offline/batch job.
For real applications, the best way to set expectations is to define two paths and measure them separately. The online path is “query embedding + vector search”: embed a short query (often tens of tokens) and search top-k in your vector store. The offline path is “document ingestion”: chunk documents, batch embed (hundreds or thousands of chunks), and upsert vectors. Offline latency is about throughput (chunks/sec) and job completion time; online latency is about p95/p99. If you’re seeing slow online performance, the usual culprits are: embedding long user inputs, synchronous re-embedding of documents during requests, unbounded retries, or saturating your own worker pool (queueing). If you’re seeing slow offline jobs, batching, concurrency limits, and idempotent retries matter more than micro-optimizing your code.
Vector search latency is the other half of the story, and it’s tunable. A vector database such as Milvus or Zilliz Cloud offers approximate indexes whose parameters trade recall for speed. Higher recall settings can increase latency; tighter latency settings can reduce recall. Filtering can also change performance depending on how you partition data (for example, per-tenant partitions can make filtered search faster by reducing the candidate set). The practical guidance is to benchmark with your real corpus shape: choose a chunk size, build an index, then measure (a) embedding time for typical query lengths, (b) search time for top-k at your target filters, and © combined p95/p99 under concurrency. That gives you a defensible latency target and a clear knob map (chunking, batching, ANN parameters, and concurrency control) to hit it.
For more information, click here: https://zilliz.com/ai-models/voyage-large-2