[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312181#comment-14312181 ]
Maxim Ivanov commented on KAFKA-1933: ------------------------------------- Updated patch, now it correctly calculates offset ranges. On next iteration I'll clenup spaghetti code in Log.append so it is more clear what is happening > Fine-grained locking in log append > ---------------------------------- > > Key: KAFKA-1933 > URL: https://issues.apache.org/jira/browse/KAFKA-1933 > Project: Kafka > Issue Type: Improvement > Components: log > Reporter: Maxim Ivanov > Assignee: Jay Kreps > Priority: Minor > Fix For: 0.8.2 > > Attachments: KAFKA-1933.patch, KAFKA-1933_2015-02-09_12:27:06.patch > > > This patch adds finer locking when appending to log. It breaks > global append lock into 2 sequential and 1 parallel phase. > Basic idea is to allow every thread to "reserve" offsets in non > overlapping ranges, then do compression in parallel and then > "commit" write to log in the same order offsets where reserved. > On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following > performance boost: > LZ4: 7.2 sec -> 4.2 sec > Gzip: 62.3 sec -> 26.9 sec > Kafka was configured to run 4 IO threads, data was pushed using 5 netcat > instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in > total, 82180 messages in total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)