Configure Message Storage with Docker Compose or Helm

Milvus uses Pulsar or Kafka for managing logs of recent changes, outputting stream logs, and providing log subscriptions. Pulsar is the default message storage system. This topic introduces how to configure message storage with Docker Compose or Helm.

You can configure Pulsar with Docker Compose or on K8s and configure Kafka on K8s.

Configure Pulsar with Docker Compose

1. Configure Pulsar

To configure Pulsar with Docker Compose, provide your values for the pulsar section in the milvus.yaml file on the milvus/configs path.

pulsar:
  address: localhost # Address of pulsar
  port: 6650 # Port of pulsar
  maxMessageSize: 5242880 # 5 * 1024 * 1024 Bytes, Maximum size of each message in pulsar.

See Pulsar-related configurations for more information.

2. Run Milvus

Run the following command to start Milvus that uses the Pulsar configurations.

docker-compose up

Configure Pulsar on K8s

Configure Pulsar on K8s

For Milvus clusters on K8s, you can configure Pulsar in the same command that starts Milvus. Alternatively, you can configure Pulsar using the values.yml file on the /charts/milvus path in the milvus-helm repository before you start Milvus.

The following table lists the keys for configuring Pulsar in the YAML file.

Using the YAML file

1. Configure the pulsar section in the values.yaml file.

pulsar:
  enabled: false

2. Configure the externalPulsar section using your values in the values.yaml file.

externalPulsar:
  enabled: true
  host: <your_pulsar_localhost>
  port: <your_pulsar_port>

3. After configuring the preceding sections and saving the values.yaml file, run the following command to install Milvus that uses the Pulsar configurations.

helm install <your_release_name> milvus/milvus -f values.yaml

Using a command

To install Milvus and configure Pulsar, run the following command using your values.

helm install <your_release_name> milvus/milvus --set cluster.enabled=true --set externalPulsar.enabled=true --set externalPulsar.host='<your_pulsar_endpoint>' --set externalPulsar.port=<your_pulsar_port>

Configure Kafka on K8s

For Milvus clusters on K8s, you can configure Kafka in the same command that starts Milvus. Alternatively, you can configure Kafka using the values.yml file on the /charts/milvus path in the milvus-helm repository before you start Milvus.

The following table lists the keys for configuring Kafka in the YAML file.

Using the YAML file

1. Configure the kafka section in the values.yaml file.

kafka:
  enabled: false
  name: kafka
  replicaCount: 3
  image:
    repository: bitnami/kafka
    tag: 3.1.0-debian-10-r52

2. Configure the externalKafka section using your values in the values.yaml file.

externalKafka:
  enabled: false
  brokerList: <your_kafka_brokerlist>

3. After configuring the preceding sections and saving the values.yaml file, run the following command to install Milvus that uses the Kafka configurations.

helm install <your_release_name> milvus/milvus -f values.yaml

Using a command

To install Milvus and configure Kafka, run the following command using your values.

helm install <your_release_name> milvus/milvus --set cluster.enabled=true --set externalKafka.enabled=true --set externalKafka.brokerlist='<your_kafka_brokerlist>'

What’s next

Learn how to configure other Milvus dependencies with Docker Compose or Helm:

• Configure Object Storage with Docker Compose or Helm
• Configure Meta Storage with Docker Compose or Helm