[ https://issues.apache.org/jira/browse/HADOOP-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537381 ]
Mukund Madhugiri commented on HADOOP-1942: ------------------------------------------ Please ignore my previous comment. The run did not actually go thru with 120 threads. It was run with 40 threads as well. Here is new data from a 100 node benchmark run: #1. TPS without this fix, 40 threads CreateWrite: 1023 OpenRead: 46728 Rename: 1169 Delete: 962 #2. TPS with this fix, 40 threads CreateWrite: 3533 (3 times better than #1) OpenRead: 43243 Rename: 9090 (7 times better than #1) Delete: 7142 (7 times better than #1) #3. TPS with this fix, 120 threads CreateWrite: 4138 (4 times better than #1) OpenRead: 50251 Rename: 10097 (8 times better than #1) Delete: 7210 (7 times better than #1) > Increase the concurrency of transaction logging to edits log > ------------------------------------------------------------ > > Key: HADOOP-1942 > URL: https://issues.apache.org/jira/browse/HADOOP-1942 > Project: Hadoop > Issue Type: Improvement > Components: dfs > Reporter: dhruba borthakur > Assignee: dhruba borthakur > Priority: Blocker > Fix For: 0.15.0 > > Attachments: transactionLogSync.patch, transactionLogSync2.patch, > transactionLogSync3.patch, transactionLogSync4.patch, > transactionLogSync5.patch, transactionLogSync6.patch, > transactionLogSync8.patch, transactionLogSync9.patch > > > For some typical workloads, the throughput of the namenode is bottlenecked by > the rate of transactions that are being logged into tghe edits log. In the > current code, a batching scheme implies that all transactions do not have to > incur a sync of the edits log to disk. However, the existing batch-ing scheme > can be improved. > One option is to keep two buffers associated with edits file. Threads write > to the primary buffer while holding the FSNamesystem lock. Then the thread > release the FSNamesystem lock, acquires a new lock called the syncLock, swaps > buffers, and flushes the old buffer to the persistent store. Since the > buffers are swapped, new transactions continue to get logged into the new > buffer. (Of course, the new transactions cannot complete before this new > buffer is sync-ed). > This approach does a better job of batching syncs to disk, thus improving > performance. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.