항등성 있는 일괄 가져오기
대량 가져오기 요청은 성공했음에도 응답이 손실될 수 있습니다. 즉, 호출이 시간 초과되거나, 연결이 끊기거나, 반환된 jobId 를 저장하기 전에 오케스트레이터가 종료될 수 있습니다. “완료되지 않음”과 “완료되었으나 응답 손실”을 구분할 수 없습니다. 재시도하면 두 번째 작업이 시작되어 모든 행이 두 번 처리되며, 재시도하지 않으면 가져오기 작업이 손실될 수 있습니다.
이멱포텐트 키를 사용하면 이러한 선택의 여지가 사라집니다. 요청에 이멱포텐트 키를 태그로 지정하면, 동일한 키를 가진 재시도 요청은 두 번째 작업을 생성하는 대신 원래의 ` jobId `를 반환합니다.
이멱포텐트 일괄 가져오기를 사용하려면 Milvus 3.0.2 이상 또는 2.6 계열의 2.6.24 이상이 필요합니다. 이전 버전의 서버는 헤더를 무시하며, 재시도 여부와 관계없이 모든 요청이 새로운 작업을 시작합니다.
키 전송
키는 요청 본문이 아닌 Idempotency-Key HTTP 헤더로 전송됩니다.
curl -X POST "http://localhost:19530/v2/vectordb/jobs/import/create" \
-H "Authorization: Bearer root:Milvus" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: nightly-2026-09-07-batch-3" \
-d '{"collectionName": "events", "files": [["events/2026-09-07/part-0.parquet"]]}'
resp = bulk_import( url="http://localhost:19530", collection_name="events", files=[[“events/2026-09-07/part-0.parquet”]], headers={IDEMPOTENCY_KEY_HEADER: "nightly-2026-09-07-batch-3"}, ) resp.json()[“data”][“jobId”] # 재시도할 때마다 동일한 ID
`IDEMPOTENCY_KEY_HEADER` is `"Idempotency-Key"`; the raw string works too. The `headers` parameter and `IDEMPOTENCY_KEY_HEADER` require pymilvus v3.0.3 or later on the 3.x line, or v2.6.18 or later on the 2.6 line. On an older pymilvus, the `import` line above fails with an `ImportError` at the `IDEMPOTENCY_KEY_HEADER` lookup, and passing `headers` to `bulk_import` has undefined behavior — the call may error, or the import may be submitted without the key.
Poll `/v2/vectordb/jobs/import/describe` with the returned `jobId` as usual. A retry that arrives while the original is still being registered waits for it, so a retry never observes a half-registered job.
## Choosing a key
- **One key per logical request.** Name the unit of work, such as `orders-2026-09-07-batch-3`, or generate a UUID and store it with the work item before you send the request. Include an attempt number if you may need to re-run the same work.
- **Retry with the same key.** Minting a new key on a retry is what duplicates data. The exceptions are listed under [When to send a new key](#when-to-send-a-new-key).
- **Never reuse a key for different files.** Milvus does not compare the two requests. You get the original `jobId` back, and the new files are not imported.
- **Keep it short and printable.** Printable ASCII, at most 256 bytes by default (`streaming.idempotency.maxKeyLength`). Anything else is rejected with error `1100`.
- **Do not put secrets in it.** The key is stored with your data.
## How long a key is remembered
The key is scoped to the collection by its internal ID, so renaming the collection does not break a retry, as long as the retry names the collection by its new name. Dropping and recreating the collection under the same name does break it, and a fresh job is the right answer there.
Milvus remembers an import key for about a day. Treat that as nominal rather than guaranteed: heavy DDL or import traffic shortens it, and a coordinator restart lengthens it. A retry after the key is forgotten starts a fresh job.
## When to send a new key
Two cases, and only two:
- **The original job failed or was aborted.** Describe returns `state: Failed` with a reason. Retrying the key returns that same `jobId` for as long as the key is remembered. Fix the cause, then send a new key.
- **The original job was cleaned up.** Milvus keeps finished jobs for `dataCoord.import.taskRetention` (48 hours by default). If that expires while the key is still remembered, the retry returns the old `jobId`, and describe answers with the error `import job does not exist` instead of a job state. That job may have succeeded. Confirm the data is missing before you send a new key.
A retry that is rejected, for example because the import job limit is full, is not one of these cases. Retry it later with the same key.
## Configuration
Four parameters bound the behavior on this page. None of them needs changing for the key to work.
| Parameter | Default | What it does |
|---|---|---|
| `streaming.walBroadcaster.tombstone.maxLifetime` | `24h` | How long an import key is remembered. Not present in the default `milvus.yaml`; add it explicitly to change it. The same store also rejects duplicate DDL submissions, so do not lower it just to shorten the import window. |
| `streaming.walBroadcaster.tombstone.maxCount` | `8192` | How many keys and DDL records the same store holds before the oldest are dropped, whatever their age. This is what lets heavy DDL or import traffic forget a key before `maxLifetime`. Not present in the default `milvus.yaml`. |
| `dataCoord.import.taskRetention` | `172800` (48 hours) since 2.6.24 and 3.0.2; `10800` before | How long a finished import job stays queryable. Keep it at least twice `maxLifetime`. |
| `streaming.idempotency.maxKeyLength` | `256` | The maximum key length in bytes. A longer key is rejected with error `1100`. `0` disables the bound. Not present in the default `milvus.yaml`. |
The `milvus.yaml` shipped before 2.6.24 and 3.0.2 sets `taskRetention: 10800` explicitly, and an upgrade does not override a value in your file. If you upgraded with that file, raise `taskRetention` to at least twice `maxLifetime`. The new value applies only to jobs that finish after the change.
