From Milvus 2.3.x
This guide provides a comprehensive, step-by-step process for migrating data from Milvus 2.3.x to Milvus 2.3.x or above.
Prerequisites
- Software versions:
- Source Milvus: 2.3.0+ (The tool uses the iterator to fetch source collection data, requiring source Milvus to be version 2.3.0 or above.)
- Target Milvus: 2.3.0+
- Required tools:
- Milvus-migration tool. For installation details, refer to Install Migration Tool.
- Data preparation:
- Ensure that the source Milvus collection is loaded and ready for data export.
- If the target Milvus does not contain a collection corresponding to the source collection, the milvus-migration tool will automatically create it. Note that after migration, the target collection will not be indexed, and you must manually index the collection afterward.
Configure the migration file
Save the example migration config file as migration.yaml
and modify the configs based on your actual conditions. You are free to put the config file in any local directory.
dumper:
worker:
workMode: milvus2x
reader:
bufferSize: 500
meta:
mode: config
version: 2.3.0
collection: src_table_name
source:
milvus2x:
endpoint: {milvus2x_domain}:{milvus2x_port}
username: xxxx
password: xxxxx
target:
milvus2x:
endpoint: {milvus2x_domain}:{milvus2x_port}
username: xxxx
password: xxxxx
The following table describes the parameters in the example config file. For more information, refer to Milvus Migration: Milvus2.x to Milvus2.x.
dumper
Parameter Description dumper.worker.workMode
The operational mode of the migration job. Set to milvus2x when migrating from Milvus 2.x. dumper.worker.reader.bufferSize
Buffer size to read from Milvus 2.x in each batch. meta
Parameter Description meta.mode
Specifies where the meta file is read from. Set to config, indicating that the meta config can be obtained from this migration.yaml file. meta.version
Source Milvus version. Set to 2.3.0 or above. meta.collection
Source collection name. source
Parameter Description source.milvus2x.endpoint
Address of the source Milvus server. source.milvus2x.username
Username for the source Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to Enable Authentication. source.milvus2x.password
Password for the source Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to Enable Authentication. target
Parameter Description target.milvus2x.endpoint
Address of the target Milvus server. target.milvus2x.username
Username for the target Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to Enable Authentication. target.milvus2x.password
Password for the target Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to Enable Authentication.
Start the migration task
You have two options to start the migration task - using CLI or making API requests. Choose the one that best fits your needs.
Option 1: Using CLI
Start the migration task with the following command. Replace {YourConfigFilePath}
with the local directory where the config file migration.yaml
resides.
./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml
Monitor the logs for progress updates. Successful migration logs should include entries like:
[INFO] [migration/milvus2x_starter.go:79] ["=================>JobProcess!"] [Percent=100]
[INFO] [migration/milvus2x_starter.go:27] ["[Starter] migration Milvus2x to Milvus2x finish!!!"] [Cost=94.877717375]
[INFO] [starter/starter.go:109] ["[Starter] Migration Success!"] [Cost=94.878243583]
Option 2: Making API requests
You can also use the Restful API to execute the migration. Start the API server with:
./milvus-migration server run -p 8080
Once the server starts successfully, place the migration.yaml
file in the configs/
directory of the project and start the migration using:
curl -XPOST http://localhost:8080/api/v1/start
Verify the result
After the migration task is completed, use Attu to view the number of entities migrated. Additionally, you can create indexes and load collections in Attu. For more information, refer to Attu and get_collection_stats().
Additional configuration options
In addition to the basic configurations mentioned above, you can also add additional settings based on your specific requirements.
Selective field migration: If you need to migrate only specific fields in a collection rather than all fields, specify the fields to be migrated in the
meta
section of themigration.yaml
file.meta: fields: - name: id - name: title_vector - name: reading_time
Custom target collection: To customize the properties of the target collection, add the related configurations in the
meta
section of themigration.yaml
file.meta: milvus: collection: target_collection_name shardNum: 2 closeDynamicField: false consistencyLevel: Customized
For detailed information, refer to Milvus Migration: Milvus2.x to Milvus2.x.