[
https://issues.apache.org/jira/browse/KUDU-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wxmimperio updated KUDU-2995:
-----------------------------
Description:
Kudu version： kudu 1.7.0-cdh5.16.2
System memary pre node：256G
4 SSDs per machine：512G
Three Master nodes and three Tserver nodes.
{code:java}
// Master config
--fs_wal_dir=/mnt/disk1/kudu/var/wal
--fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
--fs_metadata_dir=/mnt/disk1/kudu/var/metadata
--log_dir=/mnt/disk1/kudu/var/logs
--master_addresses=xxxx
--maintenance_manager_num_threads=2
--block_cache_capacity_mb=6144
--memory_limit_hard_bytes=34359738368
--max_log_size=40
// Tserver config
--fs_wal_dir=/mnt/disk1/kudu/var/wal
--fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
--fs_metadata_dir=/mnt/disk1/kudu/var/metadata
--log_dir=/mnt/disk1/kudu/var/logs
--tserver_master_addrs=xxxx
--block_cache_capacity_mb=6144
--memory_limit_hard_bytes=34359738368
--max_log_size=40
// Table schema
// _key is UUID for each msg
// event_time is data time
// Schema has only 15 columns
// Single message does not exceed 100Bytes
HASH (_key) PARTITIONS 3,
RANGE (event_time) (
PARTITION 2019-10-31T16:00:00.000000Z <= VALUES <
2019-11-30T16:00:00.000000Z
)
{code}
I write a project to write data to kudu.
Whether manual or automatic flush mode write speed is only 6MB/s.
I think SSD should be more than this speed, and the network and memory have not
reached the bottleneck.
Is this the normal level of kudu writing? How to tuning?
was:
Kudu version： kudu 1.7.0-cdh5.16.2
System memary pre node：256G
4 SSDs per machine：512G
Three Master nodes and three Tserver nodes.
{code:java}
// Master config
--fs_wal_dir=/mnt/disk1/kudu/var/wal
--fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
--fs_metadata_dir=/mnt/disk1/kudu/var/metadata
--log_dir=/mnt/disk1/kudu/var/logs
--master_addresses=xxxx
--maintenance_manager_num_threads=2
--block_cache_capacity_mb=6144
--memory_limit_hard_bytes=34359738368
--max_log_size=40
// Tserver config
--fs_wal_dir=/mnt/disk1/kudu/var/wal
--fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
--fs_metadata_dir=/mnt/disk1/kudu/var/metadata
--log_dir=/mnt/disk1/kudu/var/logs
--tserver_master_addrs=xxxx
--block_cache_capacity_mb=6144
--memory_limit_hard_bytes=34359738368
--max_log_size=40
// Table schema
// _key is UUID for each msg
// event_time is data time
// Schema has only 15 columns
// Single message does not exceed 100Bytes
HASH (_key) PARTITIONS 3,
RANGE (event_time) (
PARTITION 2019-10-31T16:00:00.000000Z <= VALUES <
2019-11-30T16:00:00.000000Z
)
{code}
I write a project to write data to kudu.
Whether manual or automatic flush mode write speed is only 6M/s.
I think SSD should be more than this speed, and the network and memory have not
reached the bottleneck.
Is this the normal level of kudu writing? How to tuning?
> Tuning kudu write performance
> -----------------------------
>
> Key: KUDU-2995
> URL: https://issues.apache.org/jira/browse/KUDU-2995
> Project: Kudu
> Issue Type: Test
> Components: client, java
> Affects Versions: 1.7.0
> Environment: kudu 1.7.0-cdh5.16.2
> Reporter: wxmimperio
> Priority: Major
> Labels: performance
>
> Kudu version： kudu 1.7.0-cdh5.16.2
> System memary pre node：256G
> 4 SSDs per machine：512G
> Three Master nodes and three Tserver nodes.
> {code:java}
> // Master config
> --fs_wal_dir=/mnt/disk1/kudu/var/wal
> --fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
> --fs_metadata_dir=/mnt/disk1/kudu/var/metadata
> --log_dir=/mnt/disk1/kudu/var/logs
> --master_addresses=xxxx
> --maintenance_manager_num_threads=2
> --block_cache_capacity_mb=6144
> --memory_limit_hard_bytes=34359738368
> --max_log_size=40
> // Tserver config
> --fs_wal_dir=/mnt/disk1/kudu/var/wal
> --fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
> --fs_metadata_dir=/mnt/disk1/kudu/var/metadata
> --log_dir=/mnt/disk1/kudu/var/logs
> --tserver_master_addrs=xxxx
> --block_cache_capacity_mb=6144
> --memory_limit_hard_bytes=34359738368
> --max_log_size=40
> // Table schema
> // _key is UUID for each msg
> // event_time is data time
> // Schema has only 15 columns
> // Single message does not exceed 100Bytes
> HASH (_key) PARTITIONS 3,
> RANGE (event_time) (
> PARTITION 2019-10-31T16:00:00.000000Z <= VALUES <
> 2019-11-30T16:00:00.000000Z
> )
> {code}
> I write a project to write data to kudu.
> Whether manual or automatic flush mode write speed is only 6MB/s.
> I think SSD should be more than this speed, and the network and memory have
> not reached the bottleneck.
> Is this the normal level of kudu writing? How to tuning?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
