[
https://issues.apache.org/jira/browse/LOG4J2-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763847#comment-15763847
]
Vincent Tieleman commented on LOG4J2-1733:
------------------------------------------
Hi Remko,
The change was more about the relative performance improvement than the
absolute performance improvement; I am running the code on quite a slow
machine. Nevertheless I repeated the tests for you.
Here are the results from the kafka performance test
(org.apache.kafka.tools.ProducerPerformance, which I adjusted to simulate the
log4j2 code).
syncSend=true:
3195 records sent, 638.9 records/sec (0.06 MB/sec), 1.5 ms avg latency, 20.0
max latency.
3312 records sent, 662.4 records/sec (0.06 MB/sec), 1.5 ms avg latency, 19.0
max latency.
3125 records sent, 624.9 records/sec (0.06 MB/sec), 1.6 ms avg latency, 11.0
max latency.
3226 records sent, 645.1 records/sec (0.06 MB/sec), 1.5 ms avg latency, 26.0
max latency.
3246 records sent, 649.1 records/sec (0.06 MB/sec), 1.5 ms avg latency, 15.0
max latency.
3214 records sent, 642.8 records/sec (0.06 MB/sec), 1.5 ms avg latency, 9.0 max
latency.
3349 records sent, 669.7 records/sec (0.06 MB/sec), 1.5 ms avg latency, 10.0
max latency.
3381 records sent, 676.2 records/sec (0.06 MB/sec), 1.4 ms avg latency, 10.0
max latency.
3377 records sent, 675.3 records/sec (0.06 MB/sec), 1.4 ms avg latency, 20.0
max latency.
3232 records sent, 646.4 records/sec (0.06 MB/sec), 1.5 ms avg latency, 18.0
max latency.
Average: 653,08 records/sec
syncSend=false:
181109 records sent, 36221.8 records/sec (3.45 MB/sec), 14431.3 ms avg latency,
14525.0 max latency.
168299 records sent, 33659.8 records/sec (3.21 MB/sec), 14396.6 ms avg latency,
14466.0 max latency.
167567 records sent, 33513.4 records/sec (3.20 MB/sec), 14470.5 ms avg latency,
14559.0 max latency.
172996 records sent, 34599.2 records/sec (3.30 MB/sec), 14613.5 ms avg latency,
14750.0 max latency.
174094 records sent, 34811.8 records/sec (3.32 MB/sec), 14598.8 ms avg latency,
14753.0 max latency.
168970 records sent, 33794.0 records/sec (3.22 MB/sec), 14544.1 ms avg latency,
14574.0 max latency.
171532 records sent, 34306.4 records/sec (3.27 MB/sec), 14547.1 ms avg latency,
14590.0 max latency.
165920 records sent, 33184.0 records/sec (3.16 MB/sec), 14723.5 ms avg latency,
14819.0 max latency.
164639 records sent, 32927.8 records/sec (3.14 MB/sec), 14846.5 ms avg latency,
14988.0 max latency.
166408 records sent, 33281.6 records/sec (3.17 MB/sec), 14972.8 ms avg latency,
15077.0 max latency.
Average: 34029,98 records/sec
In other words, approximately 52x faster, which is what you would expect to see
with log4j2 as well. However, I did not repeat the test with log4j2.
What I did test, is the application I am using log4j2 and Kafka for. There the
gain is smaller, as the application itself is the bottleneck once syncSend has
been set to false. The performance increase I saw in the application was
approximately 32x for around 830000 messages (from 1350 seconds down to 42
seconds).
Hope this helps!
> Add SyncSend attribute to KafkaAppender (as in KafkaLog4jAppender)
> ------------------------------------------------------------------
>
> Key: LOG4J2-1733
> URL: https://issues.apache.org/jira/browse/LOG4J2-1733
> Project: Log4j 2
> Issue Type: Improvement
> Components: Appenders
> Affects Versions: 2.5
> Environment: Kafka 0.10.1.0
> Reporter: Vincent Tieleman
> Assignee: Mikael Ståldal
> Fix For: 2.8
>
>
> The KafkaLog4jAppender (shipped with Kafka and usable with log4j), has a
> syncsend property that improves performance significantly by not blocking a
> send to Kafka. I've tried many other configuration settings and setups, but
> find that none of these approaches improved the logging performance to Kafka
> as significantly as setting the syncsend property to false.
> Unfortunately, the syncsend property is not supported by the KafkaAppender
> shipped with log4j2 and the KafkaLog4jAppender only supports log4j, so I am
> stuck with forking the code and making the change myself.
> Could you please introduce this property in the KafkaAppender in log4j2?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
