milvus-logo
LFAI
Home
  • Milvus Migration

From Faiss

This guide provides a comprehensive, step-by-step process for migrating data from Faiss to Milvus 2.x. By following this guide, you will be able to efficiently transfer your data, leveraging Milvus 2.x advanced features and improved performance.

Prerequisites

Configure the migration

Save the example migration config file as migration.yaml and modify the configs based on your actual conditions. You are free to put the config file in any local directory.

dumper: # configs for the migration job.
  worker:
    limit: 2
    workMode: faiss    # operational mode of the migration job.
    reader:
      bufferSize: 1024
    writer:
      bufferSize: 1024
loader:
  worker:
    limit: 2
source: # configs for the source Faiss index.
  mode: local
  local:
    faissFile: ./testfiles/faiss/faiss_ivf_flat.index

target: # configs for the target Milvus collection.
  create:
    collection:
      name: test1w
      shardsNums: 2
      dim: 256
      metricType: L2

  mode: remote
  remote:
    outputDir: testfiles/output/
    cloud: aws
    endpoint: 0.0.0.0:9000
    region: ap-southeast-1
    bucket: a-bucket
    ak: minioadmin
    sk: minioadmin
    useIAM: false
    useSSL: false
    checkBucket: true
  milvus2x:
    endpoint: localhost:19530
    username: xxxxx
    password: xxxxx

The following table describes the parameters in the example config file. For a full list of configs, refer to Milvus Migration: Faiss to Milvus 2.x.

  • dumper

    ParameterDescription
    dumper.worker.limitThe concurrency of dumper threads.
    dumper.worker.workModeThe operational mode of the migration job. Set to faiss when migrating from Faiss indexes.
    dumper.worker.reader.bufferSizeBuffer size to read from Faiss in each batch. Unit: KB.
    dumper.worker.writer.bufferSizeBuffer size to write to Milvus in each batch. Unit: KB.
  • loader

    ParameterDescription
    loader.worker.limitThe concurrency of loader threads.
  • source

    ParameterDescription
    source.modeSpecifies where the source files are read from. Valid values:
    - local: reads files from a local disk.
    - remote: reads files from remote storage.
    source.local.faissFileThe directory path where the source files are located. For example, /db/faiss.index.
  • target

    ParameterDescription
    target.create.collection.nameName of the Milvus collection.
    target.create.collection.shardsNumsNumber of shards to be created in the collection. For more information on shards, refer to Terminology.
    target.create.collection.dimDimension of the vector field.
    target.create.collection.metricTypeMetric type used to measure similarities between vectors. For more information, refer to Terminology.
    target.modeStorage location for dumped files. Valid values:
    - local: Store dumped files on local disks.
    - remote: Store dumped files on object storage.
    target.remote.outputDirOutput directory path in the cloud storage bucket.
    target.remote.cloudCloud storage service provider. Example values: aws, gcp, azure.
    target.remote.endpointEndpoint of Milvus 2.x storage.
    target.remote.regionCloud storage region. It can be any value if you use local MinIO.
    target.remote.bucketBucket name for storing data. The value must be the same as the config in Milvus 2.x. For more information, refer to System Configurations.
    target.remote.akAccess key for Milvus 2.x storage.
    target.remote.skSecret key for Milvus 2.x storage.
    target.remote.useIAMWhether to use an IAM Role for connection.
    target.remote.useSSLWhether to enable SSL when connecting to Milvus 2.x. For more information, refer to Encryption in Transit.
    target.remote.checkBucketWhether to check if the specified bucket exists in object storage.
    target.milvus2x.endpointAddress of the target Milvus server.
    target.milvus2x.usernameUsername for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to Enable Authentication.
    target.milvus2x.passwordPassword for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to Enable Authentication.

Start the migration task

  1. Start the migration task with the following command. Replace {YourConfigFilePath} with the local directory where the config file migration.yaml resides.

    ./milvus-migration  dump  --config=/{YourConfigFilePath}/migration.yaml
    

    The command above converts the Faiss index data into NumPy files, and then uses the bulkInsert operation to write the data to the target bucket.

  2. Once NumPy files are generated, import these files into Milvus 2.x with the following command. Replace {YourConfigFilePath} with the local directory where the config file migration.yaml resides.

    ./milvus-migration  load  --config=/{YourConfigFilePath}/migration.yaml
    

Verify the result

Once the migration task is executed, you can make API calls or use Attu to view the number of entities migrated. For more information, refer to Attu and get_collection_stats().

Try Managed Milvus for Free

Zilliz Cloud is hassle-free, powered by Milvus and 10x faster.

Get Started
Feedback

Was this page helpful?