Deploy CDC Server

This guide provides a step-by-step process for deploying a Milvus-CDC server.

Prerequisites

Ensure the following conditions are met before deploying a Milvus-CDC server:

Milvus Instances: Both the source Milvus and at least one target Milvus should be deployed and operational.
- Both the source and target Milvus versions must be 2.3.2 or higher, preferably 2.4.x. We recommend uisng the same version for the source and target Milvus to ensure compatibility.
- Set the common.ttMsgEnabled configuration of the target Milvus to false.
- Configure the source and target Milvus with distinct meta and message storage settings to prevent conflicts. For instance, avoid using the same etcd and rootPath configurations, as well as identical Pulsar services and chanNamePrefix in multiple Milvus instances.
Metastore: Have an etcd or MySQL database ready for the Milvus-CDC metastore.

Steps

Obtain the Milvus-CDC config file

Clone the Milvus-CDC repo and navigate to the milvus-cdc/server/configs directory to access the cdc.yaml config file.

git clone https://github.com/zilliztech/milvus-cdc.git

cd milvus-cdc/server/configs

Edit the config file

In the milvus-cdc/server/configs directory, modify the cdc.yaml file to customize configurations related to the Milvus-CDC metastore and connection details of the source Milvus.

Metastore Configuration:
- metaStoreConfig.storeType: Type of metastore for Milvus-CDC. Possible values are etcd or mysql.
- metaStoreConfig.etcdEndpoints: Address for connecting to the etcd of Milvus-CDC. Required if storeType is set to etcd.
- metaStoreConfig.mysqlSourceUrl: Connection address of the MySQL database for the Milvus-CDC server. Required if storeType is set to mysql.
- metaStoreConfig.rootPath: Root path of the Milvus-CDC metastore. This configuration enables multi-tenancy, allowing multiple CDC services to utilize the same etcd or MySQL instance while achieving isolation through different root paths.
Example configuration:
```
# cdc meta data config
metaStoreConfig:
  # the metastore type, available value: etcd, mysql
  storeType: etcd
  # etcd address
  etcdEndpoints:
    - localhost:2379
  # mysql connection address
  # mysqlSourceUrl: root:root@tcp(127.0.0.1:3306)/milvus-cdc?charset=utf8
  # meta data prefix, if multiple cdc services use the same store service, you can set different rootPaths to achieve multi-tenancy
  rootPath: cdc
```
Source Milvus Configuration:

Specify the connection details of the source Milvus, including etcd and message storage, to establish a connection between the Milvus-CDC server and the source Milvus.
- sourceConfig.etcdAddress: Address for connecting to the etcd of the source Milvus. For more information, refer to etcd-related Configurations.
- sourceConfig.etcdRootPath: Root prefix of the key where the source Milvus stores data in etcd. The value may vary based on the deployment method of the Milvus instance:
  - Helm or Docker Compose: Defaults to by-dev.
  - Operator: Defaults to <release_name>.
- sourceConfig.pulsar: Pulsar configurations for the source Milvus. If the source Milvus uses Kafka for message storage, remove all Pulsar-related configurations. For more information, refer to Pulsar-related Configurations.
- sourceConfig.kafka.address: Kafka address for the source Milvus. Uncomment this configuration if the source Milvus uses Kafka for message storage. For more information, refer to Kafka-related Configurations.

Example configuration:

Compile the Milvus-CDC server

After saving the cdc.yaml file, navigate to the milvus-cdc directory and run one of the following commands to compile the server:

For a binary file:
```
make build
```
For a Docker image:
```
bash build_image.sh
```
For a Docker image, mount the compiled file to /app/server/configs/cdc.yaml within the container.

Start the server

Using the binary

Navigate to the directory containing the milvus-cdc binary and the configs directory with the cdc.yaml file, then start the server:

# dir tree
.
├── milvus-cdc # build from source code or download from release page
├── configs
│   └── cdc.yaml # config for cdc and source milvus

# start milvus cdc
./milvus-cdc server

Using Docker Compose:
```
docker-compose up -d
```