[ https://issues.apache.org/jira/browse/HDDS-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Glen Geng updated HDDS-4228: ---------------------------- Labels: (was: pull-request-available) > ALLOCATE_BLOCK of scm audit log miss num > ---------------------------------------- > > Key: HDDS-4228 > URL: https://issues.apache.org/jira/browse/HDDS-4228 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Reporter: Glen Geng > Assignee: Glen Geng > Priority: Minor > > Teragen reported to be slow with low number of mappers compared to HDFS. > In my test (one pipeline, 3 yarn nodes) 10 g teragen with HDFS was ~3 mins > but with Ozone it was 6 mins. It could be fixed with using more mappers, but > when I investigated the execution I found a few problems reagrding to the > BufferPool management. > 1. IncrementalChunkBuffer is slow and it might not be required as BufferPool > itself is incremental > 2. For each write operation the bufferPool.allocateBufferIfNeeded is called > which can be a slow operation (positions should be calculated). > 3. There is no explicit support for write(byte) operations > In the flamegraph it's clearly visible that with low number of mappers the > client is busy with buffer operations. After the patch the rpc call and the > checksum calculation give the majority of the time. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org