milvus-logo
LFAI
Home
  • Administration Guide

Upgrade Milvus Cluster with Helm Chart

This guide describes how to upgrade your Milvus cluster with Milvus Helm charts.

Check Milvus Helm Chart

Run the following commands to check new Milvus versions.

$ helm repo update
$ helm search repo zilliztech/milvus --versions

The Milvus Helm Charts repo at https://milvus-io.github.io/milvus-helm/ has been archived and you can get further updates from https://zilliztech.github.io/milvus-helm/ as follows:

helm repo add zilliztech https://zilliztech.github.io/milvus-helm
helm repo update
# upgrade existing helm release
helm upgrade my-release zilliztech/milvus

The archived repo is still available for the charts up to 4.0.31. For later releases, use the new repo instead.

NAME                    CHART VERSION   APP VERSION             DESCRIPTION                                       
zilliztech/milvus       4.0.34          2.2.14                  Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.33          2.2.13                  Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.32          2.2.13                  Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.31          2.2.13                  Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.30          2.2.13                  Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.29          2.2.12                  Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.28          2.2.11                  Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.27          2.2.11                  Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.26          2.2.11                  Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.25          2.2.10                  Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.24          2.2.9                   Milvus is an open-source vector database built ...
zilliztech/milvus       4.0.23          2.2.9                   Milvus is an open-source vector database built ...

You can choose the upgrade path for your Milvus as follows:

Conduct a rolling upgrade

Since Milvus 2.2.3, you can configure Milvus coordinators to work in active-standby mode and enable the rolling upgrade feature for them, so that Milvus can respond to incoming requests during the coordinator upgrades. In previous releases, coordinators are to be removed and then created during an upgrade, which may introduce certain downtime of the service.

Rolling upgrades requires coordinators to work in active-standby mode. You can use the script we provide to configure the coordinators to work in active-standby mode and start the rolling upgrade.

Based on the rolling update capabilities provided by Kubernetes, the above script enforces an ordered update of the deployments according to their dependencies. In addition, Milvus implements a mechanism to ensure that its components remain compatible with those depending on them during the upgrade, significantly reducing potential service downtime.

The script applies only to the upgrade of Milvus installed with Helm. The following table lists the command flags available in the scripts.

ParametersDescriptionDefault valueRequired
iMilvus instance nameNoneTrue
nNamespace that Milvus is installed indefaultFalse
tTarget Milvus versionNoneTrue
wNew Milvus image tagmilvusdb/milvus:v2.2.3True
oOperationupdateFalse

Once you have ensured that all deployments in your Milvus instance are in their normal status. You can run the following command to upgrade the Milvus instance to 2.2.16.

sh rollingUpdate.sh -n default -i my-release -o update -t 2.2.16 -w 'milvusdb/milvus:v2.2.16'
  1. The script hard-codes the upgrade order of the deployments and cannot be changed.
  2. The script uses kubectl patch to update the deployments and kubectl rollout status to watch their status.
  3. The script uses kubectl patch to update the app.kubernetes.io/version label of the deployments to the one specified after the -t flag in the command.

Upgrade Milvus using Helm

To upgrade Milvus from a minor release before v2.2.3 to the latest, run the following commands:

helm repo update
helm upgrade my-release milvus/milvus --reuse-values --version=4.0.34 # use the helm chart version here

Use the Helm chart version in the preceding command. For details on how to obtain the Helm chart version, refer to Check the Milvus version.

Migrate the metadata

Since Milvus 2.2.0, the metadata is incompatible with that in previous releases. The following example snippets assume an upgrade from Milvus 2.1.4 to Milvus 2.2.0.

1. Check the Milvus version

Run $ helm list to check your Milvus app version. You can see the APP VERSION is 2.1.4.

NAME                NAMESPACE   REVISION    UPDATED                                 STATUS      CHART           APP VERSION    
new-release         default     1           2022-11-21 15:41:25.51539 +0800 CST     deployed    milvus-3.2.18   2.1.4 

2. Check the running pods

Run $ kubectl get pods to check the running pods. You can see the following output.

NAME                                             READY   STATUS      RESTARTS   AGE
my-release-etcd-0                               1/1     Running     0          21m
my-release-etcd-1                               1/1     Running     0          21m
my-release-etcd-2                               1/1     Running     0          21m
my-release-milvus-datacoord-664c58798d-fl75s    1/1     Running     0          21m
my-release-milvus-datanode-5f75686c55-xfg2r     1/1     Running     0          21m
my-release-milvus-indexcoord-5f98b97589-2l48r   1/1     Running     0          21m
my-release-milvus-indexnode-857b4ddf98-vmd75    1/1     Running     0          21m
my-release-milvus-proxy-6c548f787f-scspp        1/1     Running     0          21m
my-release-milvus-querycoord-c454f44cd-dwmwq    1/1     Running     0          21m
my-release-milvus-querynode-76bb4946d-lbrz6     1/1     Running     0          21m
my-release-milvus-rootcoord-7764c5b686-62msm    1/1     Running     0          21m
my-release-minio-0                              1/1     Running     0          21m
my-release-minio-1                              1/1     Running     0          21m
my-release-minio-2                              1/1     Running     0          21m
my-release-minio-3                              1/1     Running     0          21m
my-release-pulsar-bookie-0                      1/1     Running     0          21m
my-release-pulsar-bookie-1                      1/1     Running     0          21m
my-release-pulsar-bookie-2                      1/1     Running     0          21m
my-release-pulsar-bookie-init-tjxpj             0/1     Completed   0          21m
my-release-pulsar-broker-0                      1/1     Running     0          21m
my-release-pulsar-proxy-0                       1/1     Running     0          21m
my-release-pulsar-pulsar-init-c8vvc             0/1     Completed   0          21m
my-release-pulsar-recovery-0                    1/1     Running     0          21m
my-release-pulsar-zookeeper-0                   1/1     Running     0          21m
my-release-pulsar-zookeeper-1                   1/1     Running     0          20m
my-release-pulsar-zookeeper-2                   1/1     Running     0          20m

3. Check the image tag

Check the image tag for the pod my-release-milvus-proxy-6c548f787f-scspp. You can see the release of your Milvus cluster is v2.1.4.

$ kubectl get pods my-release-milvus-proxy-6c548f787f-scspp -o=jsonpath='{$.spec.containers[0].image}'
# milvusdb/milvus:v2.1.4

4. Migrate the metadata

A major change in Milvus 2.2 is the metadata structure of segment indexes. Therefore, you need to use Helm to migrate the metadata while upgrading Milvus from v2.1.x to v2.2.0. Here is a script for you to safely migrate your metadata.

This script only applies to Milvus installed on a K8s cluster. Roll back to the previous version with the rollback operation first if an error occurs during the process.

The following table lists the operations you can do for meta migration.

ParametersDescriptionDefault valueRequired
iThe Milvus instance name.NoneTrue
nThe namespace that Milvus is installed in.defaultFalse
sThe source Milvus version.NoneTrue
tThe target Milvus version.NoneTrue
rThe root path of Milvus meta.by-devFalse
wThe new Milvus image tag.milvusdb/milvus:v2.2.0False
mThe meta migration image tag.milvusdb/meta-migration:v2.2.0False
oThe meta migration operation.migrateFalse
dWhether to delete migration pod after the migration is completed.falseFalse
cThe storage class for meta migration pvc.default storage classFalse
eThe etcd enpoint used by milvus.etcd svc installed with milvusFalse

1. Migrate the metadata

  1. Download the migration script.
  2. Stop the Milvus components. Any live session in the Milvus etcd can cause a migration failure.
  3. Create a backup for the Milvus metadata.
  4. Migrate the Milvus metadata.
  5. Start Milvus components with a new image.

2. Upgrade Milvus from v2.1.x to 2.2.16

The following commands assume that you upgrade Milvus from v2.1.4 to 2.2.16. Change them to the versions that fit your needs.

  1. Specify Milvus instance name, source Milvus version, and target Milvus version.

    ./migrate.sh -i my-release -s 2.1.4 -t 2.2.16
    
  2. Specify the namespace with -n if your Milvus is not installed in the default K8s namespace.

    ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.2.16
    
  3. Specify the root path with -r if your Milvus is installed with the custom rootpath.

    ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.2.16 -r by-dev
    
  4. Specify the image tag with -w if your Milvus is installed with a custom image.

    ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.2.16 -r by-dev -w milvusdb/milvus:v2.2.16
    
  5. Set -d true if you want to automatically remove the migration pod after the migration is completed.

    ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.2.16 -w milvusdb/milvus:v2.2.16 -d true
    
  6. Rollback and migrate again if the migration fails.

    ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.2.16 -r by-dev -o rollback -w milvusdb/milvus:v2.1.1
    ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.2.16 -r by-dev -o migrate -w milvusdb/milvus:v2.2.16