Milvus
Zilliz
Home
  • Administration Guide

dataCoord-related Configurations

dataCoord.channel.watchTimeoutInterval

Description Default Value
Timeout on watching channels (in seconds). Datanode tickler update watch progress will reset timeout timer. 300

dataCoord.channel.legacyVersionWithoutRPCWatch

Description Default Value
Datanodes <= this version are considered as legacy nodes, which doesn't have rpc based watch(). This is only used during rolling upgrade where legacy nodes won't get new channels 2.4.1

dataCoord.channel.balanceSilentDuration

Description Default Value
The duration after which the channel manager start background channel balancing 300

dataCoord.channel.balanceInterval

Description Default Value
The interval with which the channel manager check dml channel balance status 360

dataCoord.channel.checkInterval

Description Default Value
The interval in seconds with which the channel manager advances channel states 1

dataCoord.channel.notifyChannelOperationTimeout

Description Default Value
Timeout notifing channel operations (in seconds). 5

dataCoord.segment.maxSize

Description Default Value
The maximum size of a segment, unit: MB. datacoord.segment.maxSize and datacoord.segment.sealProportion together determine if a segment can be sealed. 1024

dataCoord.segment.diskSegmentMaxSize

Description Default Value
Maximun size of a segment in MB for collection which has Disk index 2048

dataCoord.segment.sealProportion

Description Default Value
The minimum proportion to datacoord.segment.maxSize to seal a segment. datacoord.segment.maxSize and datacoord.segment.sealProportion together determine if a segment can be sealed. 0.12

dataCoord.segment.sealProportionJitter

Description Default Value
segment seal proportion jitter ratio, default value 0.1(10%), if seal proportion is 12%, with jitter=0.1, the actuall applied ratio will be 10.8~12% 0.1

dataCoord.segment.assignmentExpiration

Description Default Value
Expiration time of the segment assignment, unit: ms 2000

dataCoord.segment.allocLatestExpireAttempt

Description Default Value
The time attempting to alloc latest lastExpire from rootCoord after restart 200

dataCoord.segment.maxLife

Description Default Value
The max lifetime of segment in seconds, 24*60*60 86400

dataCoord.segment.maxIdleTime

Description Default Value
  • If a segment didn't accept dml records in maxIdleTime and the size of segment is greater than
  • minSizeFromIdleToSealed, Milvus will automatically seal it.
  • The max idle time of segment in seconds, 10*60.
    		•  600

    dataCoord.segment.minSizeFromIdleToSealed

    Description Default Value
    The min size in MB of segment which can be idle from sealed. 16

    dataCoord.segment.maxBinlogFileNumber

    Description Default Value
  • The max number of binlog (which is equal to the binlog file num of primary key) for one segment,
  • the segment will be sealed if the number of binlog file reaches to max value.
    		•  32

    dataCoord.segment.smallProportion

    Description Default Value
    The segment is considered as "small segment" when its # of rows is smaller than 0.5

    dataCoord.segment.compactableProportion

    Description Default Value
  • (smallProportion * segment max # of rows).
  • A compaction will happen on small segments if the segment after compaction will have
    		•  0.85

    dataCoord.segment.expansionRate

    Description Default Value
  • over (compactableProportion * segment max # of rows) rows.
  • MUST BE GREATER THAN OR EQUAL TO !!!
  • During compaction, the size of segment # of rows is able to exceed segment max # of rows by (expansionRate-1) * 100%.
    		• 1.25

    dataCoord.sealPolicy.channel.growingSegmentsMemSize

    Description Default Value
  • The size threshold in MB, if the total size of growing segments of each shard
  • exceeds this threshold, the largest growing segment will be sealed.
    		•  4096

    dataCoord.autoUpgradeSegmentIndex

    Description Default Value
    whether auto upgrade segment index to index engine's version false

    dataCoord.segmentFlushInterval

    Description Default Value
    the minimal interval duration(unit: Seconds) between flusing operation on same segment 2

    dataCoord.enableCompaction

    Description Default Value
  • Switch value to control if to enable segment compaction.
  • Compaction merges small-size segments into a large segment, and clears the entities deleted beyond the rentention duration of Time Travel.
    		•  true

    dataCoord.compaction.enableAutoCompaction

    Description Default Value
  • Switch value to control if to enable automatic segment compaction during which data coord locates and merges compactable segments in the background.
  • This configuration takes effect only when dataCoord.enableCompaction is set as true.
    		•  true

    dataCoord.compaction.taskPrioritizer

    Description Default Value
  • compaction task prioritizer, options: [default, level, mix].
  • default is FIFO.
  • level is prioritized by level: L0 compactions first, then mix compactions, then clustering compactions.
  • mix is prioritized by level: mix compactions first, then L0 compactions, then clustering compactions.
    		•  default

    dataCoord.compaction.taskQueueCapacity

    Description Default Value
    compaction task queue size 100000

    dataCoord.compaction.dropTolerance

    Description Default Value
    Compaction task will be cleaned after finish longer than this time(in seconds) 86400

    dataCoord.compaction.gcInterval

    Description Default Value
    The time interval in seconds for compaction gc 1800

    dataCoord.compaction.mix.triggerInterval

    Description Default Value
    The time interval in seconds to trigger mix compaction 60

    dataCoord.compaction.levelzero.triggerInterval

    Description Default Value
    The time interval in seconds for trigger L0 compaction 10

    dataCoord.compaction.levelzero.forceTrigger.minSize

    Description Default Value
    The minmum size in bytes to force trigger a LevelZero Compaction, default as 8MB 8388608

    dataCoord.compaction.levelzero.forceTrigger.maxSize

    Description Default Value
    The maxmum size in bytes to force trigger a LevelZero Compaction, default as 64MB 67108864

    dataCoord.compaction.levelzero.forceTrigger.deltalogMinNum

    Description Default Value
    The minimum number of deltalog files to force trigger a LevelZero Compaction 10

    dataCoord.compaction.levelzero.forceTrigger.deltalogMaxNum

    Description Default Value
    The maxmum number of deltalog files to force trigger a LevelZero Compaction, default as 30 30

    dataCoord.compaction.single.ratio.threshold

    Description Default Value
    The ratio threshold of a segment to trigger a single compaction, default as 0.2 0.2

    dataCoord.compaction.single.deltalog.maxsize

    Description Default Value
    The deltalog size of a segment to trigger a single compaction, default as 16MB 16777216

    dataCoord.compaction.single.deltalog.maxnum

    Description Default Value
    The deltalog count of a segment to trigger a compaction, default as 200 200

    dataCoord.compaction.single.expiredlog.maxsize

    Description Default Value
    The expired log size of a segment to trigger a compaction, default as 10MB 10485760

    dataCoord.compaction.clustering.enable

    Description Default Value
    Enable clustering compaction true

    dataCoord.compaction.clustering.autoEnable

    Description Default Value
    Enable auto clustering compaction false

    dataCoord.compaction.clustering.triggerInterval

    Description Default Value
    clustering compaction trigger interval in seconds 600

    dataCoord.compaction.clustering.minInterval

    Description Default Value
    The minimum interval between clustering compaction executions of one collection, to avoid redundant compaction 3600

    dataCoord.compaction.clustering.maxInterval

    Description Default Value
    If a collection haven't been clustering compacted for longer than maxInterval, force compact 259200

    dataCoord.compaction.clustering.newDataSizeThreshold

    Description Default Value
    If new data size is large than newDataSizeThreshold, execute clustering compaction 512m

    dataCoord.compaction.clustering.maxTrainSizeRatio

    Description Default Value
    max data size ratio in Kmeans train, if larger than it, will down sampling to meet this limit 0.8

    dataCoord.compaction.clustering.maxCentroidsNum

    Description Default Value
    maximum centroids number in Kmeans train 10240

    dataCoord.compaction.clustering.minCentroidsNum

    Description Default Value
    minimum centroids number in Kmeans train 16

    dataCoord.compaction.clustering.minClusterSizeRatio

    Description Default Value
    minimum cluster size / avg size in Kmeans train 0.01

    dataCoord.compaction.clustering.maxClusterSizeRatio

    Description Default Value
    maximum cluster size / avg size in Kmeans train 10

    dataCoord.compaction.clustering.maxClusterSize

    Description Default Value
    maximum cluster size in Kmeans train 5g

    dataCoord.syncSegmentsInterval

    Description Default Value
    The time interval for regularly syncing segments 300

    dataCoord.index.memSizeEstimateMultiplier

    Description Default Value
    When the memory size is not setup by index procedure, multiplier to estimate the memory size of index data 2

    dataCoord.enableGarbageCollection

    Description Default Value
    Switch value to control if to enable garbage collection to clear the discarded data in MinIO or S3 service. true

    dataCoord.gc.interval

    Description Default Value
    The interval at which data coord performs garbage collection, unit: second. 3600

    dataCoord.gc.missingTolerance

    Description Default Value
    The retention duration of the unrecorded binary log (binlog) files. Setting a reasonably large value for this parameter avoids erroneously deleting the newly created binlog files that lack metadata. Unit: second. 86400

    dataCoord.gc.dropTolerance

    Description Default Value
    The retention duration of the binlog files of the deleted segments before they are cleared, unit: second. 10800

    dataCoord.gc.removeConcurrent

    Description Default Value
    number of concurrent goroutines to remove dropped s3 objects 32

    dataCoord.gc.scanInterval

    Description Default Value
    orphan file (file on oss but has not been registered on meta) on object storage garbage collection scanning interval in hours 168

    dataCoord.brokerTimeout

    Description Default Value
    5000ms, dataCoord broker rpc timeout 5000

    dataCoord.autoBalance

    Description Default Value
    Enable auto balance true

    dataCoord.checkAutoBalanceConfigInterval

    Description Default Value
    the interval of check auto balance config 10

    dataCoord.import.filesPerPreImportTask

    Description Default Value
    The maximum number of files allowed per pre-import task. 2

    dataCoord.import.taskRetention

    Description Default Value
    The retention period in seconds for tasks in the Completed or Failed state. 10800

    dataCoord.import.maxSizeInMBPerImportTask

    Description Default Value
    To prevent generating of small segments, we will re-group imported files. This parameter represents the sum of file sizes in each group (each ImportTask). 6144

    dataCoord.import.scheduleInterval

    Description Default Value
    The interval for scheduling import, measured in seconds. 2

    dataCoord.import.checkIntervalHigh

    Description Default Value
    The interval for checking import, measured in seconds, is set to a high frequency for the import checker. 2

    dataCoord.import.checkIntervalLow

    Description Default Value
    The interval for checking import, measured in seconds, is set to a low frequency for the import checker. 120

    dataCoord.import.maxImportFileNumPerReq

    Description Default Value
    The maximum number of files allowed per single import request. 1024

    dataCoord.import.maxImportJobNum

    Description Default Value
    Maximum number of import jobs that are executing or pending. 1024

    dataCoord.import.waitForIndex

    Description Default Value
    Indicates whether the import operation waits for the completion of index building. true

    dataCoord.gracefulStopTimeout

    Description Default Value
    seconds. force stop node without graceful stop 5

    dataCoord.slot.clusteringCompactionUsage

    Description Default Value
    slot usage of clustering compaction job. 16

    dataCoord.slot.mixCompactionUsage

    Description Default Value
    slot usage of mix compaction job. 8

    dataCoord.slot.l0DeleteCompactionUsage

    Description Default Value
    slot usage of l0 compaction job. 8

    dataCoord.ip

    Description Default Value
    TCP/IP address of dataCoord. If not specified, use the first unicastable address

    dataCoord.port

    Description Default Value
    TCP port of dataCoord 13333

    dataCoord.grpc.serverMaxSendSize

    Description Default Value
    The maximum size of each RPC request that the dataCoord can send, unit: byte 536870912

    dataCoord.grpc.serverMaxRecvSize

    Description Default Value
    The maximum size of each RPC request that the dataCoord can receive, unit: byte 268435456

    dataCoord.grpc.clientMaxSendSize

    Description Default Value
    The maximum size of each RPC request that the clients on dataCoord can send, unit: byte 268435456

    dataCoord.grpc.clientMaxRecvSize

    Description Default Value
    The maximum size of each RPC request that the clients on dataCoord can receive, unit: byte 536870912
    Table of contents

    Try Managed Milvus for Free

    Zilliz Cloud is hassle-free, powered by Milvus and 10x faster.

    Get Started
    Feedback

    Was this page helpful?