dataNode-related Configurations

dataNode.dataSync.flowGraph.maxQueueLength

Description	Default Value
Maximum length of task queue in flowgraph	16

dataNode.dataSync.flowGraph.maxParallelism

Description	Default Value
Maximum number of tasks executed in parallel in the flowgraph	1024

dataNode.dataSync.maxParallelSyncMgrTasks

Description	Default Value
The max concurrent sync task number of datanode sync mgr globally	256

dataNode.dataSync.skipMode.enable

Description	Default Value
Support skip some timetick message to reduce CPU usage	true

dataNode.dataSync.skipMode.skipNum

Description	Default Value
Consume one for every n records skipped	4

dataNode.dataSync.skipMode.coldTime

Description	Default Value
Turn on skip mode after there are only timetick msg for x seconds	60

dataNode.segment.insertBufSize

Description	Default Value
The maximum size of each binlog file in a segment buffered in memory. Binlog files whose size exceeds this value are then flushed to MinIO or S3 service. Unit: Byte Setting this parameter too small causes the system to store a small amount of data too frequently. Setting it too large increases the system's demand for memory.	16777216

dataNode.segment.deleteBufBytes

Description	Default Value
Max buffer size in bytes to flush del for a single channel, default as 16MB	16777216

dataNode.segment.syncPeriod

Description	Default Value
The period to sync segments if buffer is not empty.	600

dataNode.memory.forceSyncEnable

Description	Default Value
Set true to force sync if memory usage is too high	true

dataNode.memory.forceSyncSegmentNum

Description	Default Value
number of segments to sync, segments with top largest buffer will be synced.	1

dataNode.memory.checkInterval

Description	Default Value
the interal to check datanode memory usage, in milliseconds	3000

dataNode.memory.forceSyncWatermark

Description	Default Value
memory watermark for standalone, upon reaching this watermark, segments will be synced.	0.5

dataNode.channel.workPoolSize

Description	Default Value
specify the size of global work pool of all channels if this parameter <= 0, will set it as the maximum number of CPUs that can be executing suggest to set it bigger on large collection numbers to avoid blocking	-1

dataNode.channel.updateChannelCheckpointMaxParallel

Description	Default Value
specify the size of global work pool for channel checkpoint updating if this parameter <= 0, will set it as 10	10

dataNode.channel.updateChannelCheckpointInterval

Description	Default Value
the interval duration(in seconds) for datanode to update channel checkpoint of each channel	60

dataNode.channel.updateChannelCheckpointRPCTimeout

Description	Default Value
timeout in seconds for UpdateChannelCheckpoint RPC call	20

dataNode.channel.maxChannelCheckpointsPerPRC

Description	Default Value
The maximum number of channel checkpoints per UpdateChannelCheckpoint RPC.	128

dataNode.channel.channelCheckpointUpdateTickInSeconds

Description	Default Value
The frequency, in seconds, at which the channel checkpoint updater executes updates.	10

dataNode.import.maxConcurrentTaskNum

Description	Default Value
The maximum number of import/pre-import tasks allowed to run concurrently on a datanode.	16

dataNode.import.maxImportFileSizeInGB

Description	Default Value
The maximum file size (in GB) for an import file, where an import file refers to either a Row-Based file or a set of Column-Based files.	16

dataNode.import.readBufferSizeInMB

Description	Default Value
The data block size (in MB) read from chunk manager by the datanode during import.	16

dataNode.import.maxTaskSlotNum

Description	Default Value
The maximum number of slots occupied by each import/pre-import task.	16

dataNode.compaction.levelZeroBatchMemoryRatio

Description	Default Value
The minimal memory ratio of free memory for level zero compaction executing in batch mode	0.5

dataNode.compaction.levelZeroMaxBatchSize

Description	Default Value
Max batch size refers to the max number of L1/L2 segments in a batch when executing L0 compaction. Default to -1, any value that is less than 1 means no limit. Valid range: >= 1.	-1

dataNode.compaction.useMergeSort

Description	Default Value
Whether to enable mergeSort mode when performing mixCompaction.	false

dataNode.compaction.maxSegmentMergeSort

Description	Default Value
The maximum number of segments to be merged in mergeSort mode.	30

dataNode.gracefulStopTimeout

Description	Default Value
seconds. force stop node without graceful stop	1800

dataNode.slot.slotCap

Description	Default Value
The maximum number of tasks(e.g. compaction, importing) allowed to run concurrently on a datanode	16

dataNode.clusteringCompaction.memoryBufferRatio

Description	Default Value
The ratio of memory buffer of clustering compaction. Data larger than threshold will be flushed to storage.	0.3

dataNode.clusteringCompaction.workPoolSize

Description	Default Value
worker pool size for one clustering compaction job.	8

dataNode.bloomFilterApplyParallelFactor

Description	Default Value
parallel factor when to apply pk to bloom filter, default to 4*CPU_CORE_NUM	4

dataNode.storage.deltalog

Description	Default Value
deltalog format, options: [json, parquet]	json

dataNode.ip

Description	Default Value
TCP/IP address of dataNode. If not specified, use the first unicastable address

dataNode.port

Description	Default Value
TCP port of dataNode	21124

dataNode.grpc.serverMaxSendSize

Description	Default Value
The maximum size of each RPC request that the dataNode can send, unit: byte	536870912

dataNode.grpc.serverMaxRecvSize

Description	Default Value
The maximum size of each RPC request that the dataNode can receive, unit: byte	268435456

dataNode.grpc.clientMaxSendSize

Description	Default Value
The maximum size of each RPC request that the clients on dataNode can send, unit: byte	268435456

dataNode.grpc.clientMaxRecvSize

Description	Default Value
The maximum size of each RPC request that the clients on dataNode can receive, unit: byte	536870912