Configure Message Storage with Docker Compose or Helm

Milvus uses Pulsar or Kafka for managing logs of recent changes, outputting stream logs, and providing log subscriptions. Pulsar is the default message storage system. This topic introduces how to configure message storage with Docker Compose or Helm.

You can configure Pulsar with Docker Compose or on K8s and configure Kafka on K8s.

Configure Pulsar with Docker Compose

1. Configure Pulsar

To configure Pulsar with Docker Compose, provide your values for the pulsar section in the milvus.yaml file on the milvus/configs path.

pulsar:
  address: localhost # Address of pulsar
  port: 6650 # Port of pulsar
  maxMessageSize: 5242880 # 5 * 1024 * 1024 Bytes, Maximum size of each message in pulsar.

See Pulsar-related configurations for more information.

2. Run Milvus

Run the following command to start Milvus that uses the Pulsar configurations.

docker-compose up

Configurations only take effect after Milvus starts. See Start Milvus for more information.

Configure Pulsar on K8s

For Milvus clusters on K8s, you can configure Pulsar in the same command that starts Milvus. Alternatively, you can configure Pulsar using the values.yml file on the /charts/milvus path in the milvus-helm repository before you start Milvus.

The following table lists the keys for configuring Pulsar in the YAML file.

Key	Description	Value
`pulsar.enabled`	Enables or disables Pulsar.	`true`/`false`
`externalPulsar.enabled`	Enables or disables external Pulsar.	`true`/`false`
`externalPulsar.host`	The endpoint to access external Pulsar.
`externalPulsar.port`	The port to access external Pulsar.

Using the YAML file

Configure the pulsar section in the values.yaml file.

pulsar:
  enabled: false

Configure the externalPulsar section using your values in the values.yaml file.

externalPulsar:
  enabled: true
  host: <your_pulsar_IP>
  port: <your_pulsar_port>

After configuring the preceding sections and saving the values.yaml file, run the following command to install Milvus that uses the Pulsar configurations.

helm install <your_release_name> milvus/milvus -f values.yaml

Using a command

To install Milvus and configure Pulsar, run the following command using your values.

helm install <your_release_name> milvus/milvus --set cluster.enabled=true --set pulsar.enabled=false --set externalPulsar.enabled=true --set externalPulsar.host=<your_pulsar_IP> --set externalPulsar.port=<your_pulsar_port>

Configure Kafka on K8s

For Milvus clusters on K8s, you can configure Kafka in the same command that starts Milvus. Alternatively, you can configure Kafka using the values.yml file on the /charts/milvus path in the milvus-helm repository before you start Milvus.

The following table lists the keys for configuring Pulsar in the YAML file.

Key	Description	Value
`kafka.enabled`	Enables or disables Kafka.	`true`/`false`
`externalKafka.enabled`	Enables or disables external Kafka.	`true`/`false`
`externalKafka.brokerlist`	The brokerlist to access external Kafka.

The following table lists the mandatory configurations for external Kafka. Set them in Kafka configurations.

Key	Description	Value
`max.request.size`	The maximum size of a request in bytes.	`5242880`
`message.max.bytes`	The largest record batch size allowed by Kafka.	`10485760`
`auto.create.topics.enable`	Enable auto creation of topic on the server.	`true`
`num.partitions`	The default number of log partitions per topic.	`1`

Using the YAML file

Configure the kafka section in the values.yaml file if you want to use Kafka as the message storage system.

kafka:
  enabled: true
  name: kafka
  replicaCount: 3
  image:
    repository: bitnami/kafka
    tag: 3.1.0-debian-10-r52

Configure the externalKafka section using your values in the values.yaml file if you want to use external Kafka as the message storage system.

externalKafka:
  enabled: true
  brokerList: <your_kafka_IP>:<your_kafka_port>

After configuring the preceding sections and saving the values.yaml file, run the following command to install Milvus that uses the Kafka configurations.

helm install <your_release_name> milvus/milvus -f values.yaml

Using a command

To install Milvus and configure Kafka, run the following command using your values.

helm install <your_release_name> milvus/milvus --set cluster.enabled=true --set pulsar.enabled=false --set kafka.enabled=true

To install Milvus and configure external Kafka, run the following command using your values.

helm install <your_release_name> milvus/milvus --set cluster.enabled=true --set pulsar.enabled=false --set externalKafka.enabled=true --set externalKafka.brokerlist=<your_kafka_IP>:<your_kafka_port>

What's next

Learn how to configure other Milvus dependencies with Docker Compose or Helm: