🚀 Try Zilliz Cloud, the fully managed Milvus, for free—experience 10x faster performance! Try Now>>

milvus-logo
LFAI
Home
  • Tools

Migrate Between Instances Across Buckets

This topic details the process of backing up a collection from one Milvus instance and restoring it to another, with each instance using different buckets within the same object storage.

Overview

The diagram below illustrates the backup and restore process using different buckets within the same object storage.

cross-bucket-backup-and-restore.png cross-bucket-backup-and-restore.png

Assume we have two Milvus instances, milvus_A and milvus_B, both utilizing the default MinIO storage engine for object storage. These instances use different buckets bucket_A and bucket_B within the same object storage. In this example, our goal is to complete the following tasks:

  1. Create a backup (my_backup) for collection coll in bucket_A and store the backup in bucket_B.

  2. In bucket_B, restore from the backup and name the restored collection coll_bak.

Prerequisites**

  • Ensure the milvus-backup tool is installed.

  • Familiarize yourself with configuring Milvus object storage settings. For details, refer to Object Storage.

Back up a collection from milvus_A

Step 1: Prepare configuration

Go to the directory of the milvus-backup project and create a directory named configs:

mkdir configs
cd configs

Download the backup config file backup.yaml:

wget https://raw.githubusercontent.com/zilliztech/milvus-backup/main/configs/backup.yaml

The file structure looks like this:

├── configs
│   └── backup.yaml
├── milvus-backup
└── README.md

Step 2: Edit configuration file

Modify the backup.yaml file to set the appropriate configurations for milvus_A:

  • Connection configs

    # milvus proxy address, compatible to milvus.yaml
    milvus:
      address: milvus_A
      port: 19530
      authorizationEnabled: false
      # tls mode values [0, 1, 2]
      # 0 is close, 1 is one-way authentication, 2 is two-way authentication.
      tlsMode: 0
      user: "root"
      password: "Milvus"
    
    • milvus.address: IP address or hostname of the milvus_A server.

    • milvus.port: TCP port on which Milvus server is listening (default 19530).

  • Storage configs (MinIO/S3 settings)

    # Related configuration of minio, which is responsible for data persistence for Milvus.
    minio:
      # cloudProvider: "minio" # deprecated use storageType instead
      storageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
      
      address: localhost # Address of MinIO/S3
      port: 9000   # Port of MinIO/S3
      accessKeyID: minioadmin  # accessKeyID of MinIO/S3
      secretAccessKey: minioadmin # MinIO/S3 encryption string
      useSSL: false # Access to MinIO/S3 with SSL
      useIAM: false
      iamEndpoint: ""
      
      bucketName: "bucket_A" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
      rootPath: "files" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance
    
      # only for azure
      backupAccessKeyID: minioadmin  # accessKeyID of MinIO/S3
      backupSecretAccessKey: minioadmin # MinIO/S3 encryption string
      
      backupBucketName: "bucket_B" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
      backupRootPath: "backup" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath
    
  • minio.bucketName: Name of the bucket used for data storage in milvus_A. In this example, set to bucket_A.

  • minio.rootPath: Root path within the bucket where data from milvus_A is stored. In this example, set to files.

  • minio.backupBucketName: Name of the bucket used for backup storage in milvus_B. In this example, milvus_A and milvus_B use different buckets. Therefore, set to bucket_B.

  • minio.backupRootPath: Root path within the bucket designated for storing backup files in milvus_B. In this example, set to backup.

Step 3: Create backup

Once backup.yaml is saved, create a backup named my_backup:

./milvus-backup create -c coll -n my_backup

This command creates the backup bucket_B/backup/my_backup in object storage for the collection coll.

Restore the backup to milvus_B

Step 1: Configure restoration settings

Repeat step 2 to modify configs for restoration to milvus_B, ensuring minio.bucketName is set to bucket_B.

Here’s a sample configuration:

...
# milvus proxy address, compatible to milvus.yaml
milvus:
  address: milvus_B
  port: 19530
  authorizationEnabled: false
  # tls mode values [0, 1, 2]
  # 0 is close, 1 is one-way authentication, 2 is two-way authentication.
  tlsMode: 0
  user: "root"
  password: "Milvus"
  
# Related configuration of minio, which is responsible for data persistence for Milvus.
minio:
  # cloudProvider: "minio" # deprecated use storageType instead
  storageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
  
  address: localhost # Address of MinIO/S3
  port: 9000   # Port of MinIO/S3
  accessKeyID: minioadmin  # accessKeyID of MinIO/S3
  secretAccessKey: minioadmin # MinIO/S3 encryption string
  useSSL: false # Access to MinIO/S3 with SSL
  useIAM: false
  iamEndpoint: ""
  
  bucketName: "bucket_B" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
  rootPath: "files" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance

  # only for azure
  backupAccessKeyID: minioadmin  # accessKeyID of MinIO/S3
  backupSecretAccessKey: minioadmin # MinIO/S3 encryption string
  
  backupBucketName: "bucket_B" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
  backupRootPath: "backup" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath
...

Step 2: Restore backup

Restore the backup to milvus_B:

./milvus-backup restore -c coll -n my_backup -s _bak

This command restores the backup into a new collection named coll_bak in milvus_B, with data stored in bucket_B/files/insert_log/[ID of new collection].

Try Managed Milvus for Free

Zilliz Cloud is hassle-free, powered by Milvus and 10x faster.

Get Started
Feedback

Was this page helpful?