milvus-logo
LFAI
Home
  • Administration Guide
  • Benchmarks

Configure Message Storage with Docker Compose or Helm

Milvus uses Pulsar or Kafka for managing logs of recent changes, outputting stream logs, and providing log subscriptions. Pulsar is the default message storage system. This topic introduces how to configure message storage with Docker Compose or Helm.

You can configure Pulsar with Docker Compose or on K8s and configure Kafka on K8s.

Configure Pulsar with Docker Compose

1. Configure Pulsar

To configure Pulsar with Docker Compose, provide your values for the pulsar section in the milvus.yaml file on the milvus/configs path.

pulsar:
  address: localhost # Address of pulsar
  port: 6650 # Port of pulsar
  maxMessageSize: 5242880 # 5 * 1024 * 1024 Bytes, Maximum size of each message in pulsar.

See Pulsar-related configurations for more information.

2. Run Milvus

Run the following command to start Milvus that uses the Pulsar configurations.

docker-compose up
Configurations only take effect after Milvus starts. See Start Milvus for more information.

Configure Pulsar on K8s

Configure Pulsar on K8s

For Milvus clusters on K8s, you can configure Pulsar in the same command that starts Milvus. Alternatively, you can configure Pulsar using the values.yml file on the /charts/milvus path in the milvus-helm repository before you start Milvus.

The following table lists the keys for configuring Pulsar in the YAML file.

KeyDescriptionValue
pulsar.enabledEnables or disables Pulsar.true/false
externalPulsar.enabledEnables or disables external Pulsar.true/false
externalPulsar.hostThe endpoint to access external Pulsar.
externalPulsar.portThe port to access external Pulsar.

Using the YAML file

  1. Configure the pulsar section in the values.yaml file.
pulsar:
  enabled: false
  1. Configure the externalPulsar section using your values in the values.yaml file.
externalPulsar:
  enabled: true
  host: <your_pulsar_IP>
  port: <your_pulsar_port>
  1. After configuring the preceding sections and saving the values.yaml file, run the following command to install Milvus that uses the Pulsar configurations.
helm install <your_release_name> milvus/milvus -f values.yaml

Using a command

To install Milvus and configure Pulsar, run the following command using your values.

helm install <your_release_name> milvus/milvus --set cluster.enabled=true --set pulsar.enabled=false --set externalPulsar.enabled=true --set externalPulsar.host=<your_pulsar_IP> --set externalPulsar.port=<your_pulsar_port>

Configure Kafka on K8s

For Milvus clusters on K8s, you can configure Kafka in the same command that starts Milvus. Alternatively, you can configure Kafka using the values.yml file on the /charts/milvus path in the milvus-helm repository before you start Milvus.

The following table lists the keys for configuring Pulsar in the YAML file.

KeyDescriptionValue
kafka.enabledEnables or disables Kafka.true/false
externalKafka.enabledEnables or disables external Kafka.true/false
externalKafka.brokerlistThe brokerlist to access external Kafka.

The following table lists the mandatory configurations for external Kafka. Set them in Kafka configurations.

KeyDescriptionValue
max.request.sizeThe maximum size of a request in bytes.5242880
message.max.bytesThe largest record batch size allowed by Kafka.10485760
auto.create.topics.enableEnable auto creation of topic on the server.true
num.partitionsThe default number of log partitions per topic.1

Using the YAML file

  1. Configure the kafka section in the values.yaml file if you want to use Kafka as the message storage system.
kafka:
  enabled: true
  name: kafka
  replicaCount: 3
  image:
    repository: bitnami/kafka
    tag: 3.1.0-debian-10-r52
  1. Configure the externalKafka section using your values in the values.yaml file if you want to use external Kafka as the message storage system.
externalKafka:
  enabled: true
  brokerList: <your_kafka_IP>:<your_kafka_port>
  1. After configuring the preceding sections and saving the values.yaml file, run the following command to install Milvus that uses the Kafka configurations.
helm install <your_release_name> milvus/milvus -f values.yaml

Using a command

To install Milvus and configure Kafka, run the following command using your values.

helm install <your_release_name> milvus/milvus --set cluster.enabled=true --set pulsar.enabled=false --set kafka.enabled=true

To install Milvus and configure external Kafka, run the following command using your values.

helm install <your_release_name> milvus/milvus --set cluster.enabled=true --set pulsar.enabled=false --set externalKafka.enabled=true --set externalKafka.brokerlist=<your_kafka_IP>:<your_kafka_port>

What’s next

Learn how to configure other Milvus dependencies with Docker Compose or Helm:

Table of contents
Feedback

Was this page helpful?