novosibman commented on PR #13782: URL: https://github.com/apache/kafka/pull/13782#issuecomment-1569655316
> Many thanks for the patch and the collected data! Really interesting to see the impact of this change. A few questions: > > * What storage device and file system are used in the test? In AWS config used: i3en.2xlarge with 2 x 2500 NVMe SSDs In local lab config: 2 x Samsung_SSD_860_EVO_1TB FS type: xfs The FS format had huge impact on results. Initially we used ext4 in our lab for regular testing: some of `ext4` example results:  after switched to `xfs`:  `ext4` was much worse before and during Kafka logs rolling > > * Would you have a real-life workload where the impact of this change can be quantified? The workload generated by the producer-perf-test.sh exhibits the problem the most because the segments of all replicas on the brokers start rolling at the same time. Which is why it is also interesting to assess the impact using topic-partitions which have different ingress rate and/or use segments of different sizes. We have no any real-life workload scenarios available for Kafka perf testing. Alternative workload https://github.com/AzulSystems/kafka-benchmark has slightly different rolling behavior compared to OMB: OMB results example on released kafka_2.13-3.4.0 version (using xfs):  Kafka Tussle benchmark:  # same params used: acks=1 batchSize=1048510 consumers=4 lingerMs=1 mlen=1024 partitions=100 producers=4 rf=1 targetRate=200k time=30m topics=1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
