[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378759#comment-15378759 ] Jiangjie Qin commented on KAFKA-2431: - [~benstopford] It seems I am not able to access the Google doc anymore. We recently did some performance test of SSL at LinkedIn and saw some interesting results. We would like to compare that with the tests done previously. Do you mind letting us check on the previous result? Thanks. > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ben Stopford >Priority: Blocker > Fix For: 0.9.0.0 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182739#comment-15182739 ] Ismael Juma commented on KAFKA-2431: Hi [~ssuo], an OpenSSL-based implementation is not being developed right now. See KAFKA-2561 for the information we collected. > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ben Stopford >Priority: Blocker > Fix For: 0.9.0.0 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172875#comment-15172875 ] Simon Suo commented on KAFKA-2431: -- Hi all. This is Simon, a data infra intern working on LinkedIn's Kafka team. I am currently evaluating solutions to reduce performance overhead of Kafka security features. The summary report here discusses the possibility of a optional OpenSSL implementation that may achieve 4 to 5 times speed up. Is this being developed right now? Do you have any additional benchmark data to show the potential performance gain? Let me know if you have any relevant information and time for a small discussion. I can be reached at s...@linkedin.com Best regards, Simon Suo > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ben Stopford >Priority: Blocker > Fix For: 0.9.0.0 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950644#comment-14950644 ] ASF GitHub Bot commented on KAFKA-2431: --- Github user benstopford closed the pull request at: https://github.com/apache/kafka/pull/217 > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ben Stopford >Priority: Blocker > Fix For: 0.9.0.0 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950641#comment-14950641 ] Ben Stopford commented on KAFKA-2431: - Summary report can be found here: https://docs.google.com/document/d/1HlHkjAPK9kR_lk0DF34c3NDblCYTnN6D8GENV6YI2f8 > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ben Stopford >Priority: Blocker > Fix For: 0.9.0.0 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744694#comment-14744694 ] ASF GitHub Bot commented on KAFKA-2431: --- GitHub user benstopford opened a pull request: https://github.com/apache/kafka/pull/217 KAFKA-2431: Easier Testing of SSL - Allow cipher suites to be specified relevant properties - Avoid System.exit in ProducerPerformance so this can be externally invoked - Add command line option so that a default properties can be specified in ConsumerPerformance (needed for ssl properties) You can merge this pull request into a Git repository by running: $ git pull https://github.com/benstopford/kafka ssl-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/217.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #217 commit 18fb3f0914dc284c5632522065d7e5be062ed62a Author: Ben Stopford Date: 2015-09-08T00:36:51Z KAFKA-2431: easier testing of SSL - Avoid System.exit in ProducerPerformance so this can be externally invoked - Add command line option so that a default set of properties can be speicfied in ConsumerPerformance (needed for ssl properties) - Allow cipher suites to be specified vi properties > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ben Stopford >Priority: Blocker > Fix For: 0.9.0.0 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730682#comment-14730682 ] Ben Stopford commented on KAFKA-2431: - Post SSL regression in new consumer raised here: https://issues.apache.org/jira/browse/KAFKA-2517 > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ben Stopford >Priority: Blocker > Fix For: 0.8.3 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1470#comment-1470 ] Ismael Juma commented on KAFKA-2431: Yes, definitely. We discussed that offline, but I realise now that it wasn't clear in the ticket, so thanks for mentioning that. > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ben Stopford > Fix For: 0.8.3 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709914#comment-14709914 ] Jay Kreps commented on KAFKA-2431: -- It would be good to also do the same test(s) against 0.8.2 (I'm assuming these results are from trunk for both the ssl and no ssl case). There have been a TON of changes in the network layer over all so we need to sanity check that the no SSL number is actually the true baseline. > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ben Stopford > Fix For: 0.8.3 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709852#comment-14709852 ] Ben Stopford commented on KAFKA-2431: - Progress is recorded in this doc https://docs.google.com/a/confluent.io/document/d/1svxqiHApsGtg9_dDAN60f0X-QvyYqXPKDpwx5aNI6d4/edit?usp=sharing > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ben Stopford > Fix For: 0.8.3 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance
[ https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695086#comment-14695086 ] Ismael Juma commented on KAFKA-2431: A sensible way to do this is to allow SSL/TLS to be optionally enabled with our existing performance testing tools. > Test SSL/TLS impact on performance > -- > > Key: KAFKA-2431 > URL: https://issues.apache.org/jira/browse/KAFKA-2431 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma > Fix For: 0.8.3 > > > Test new Producer and new Consumer performance with and without SSL/TLS once > the SSL/TLS branch is integrated. > The ideal scenario is that SSL/TLS would not have an impact if disabled. When > enabled, there will be some overhead (encryption and the inability to use > `SendFile`) and it will be good to quantify it. The encryption overhead is > reduced if recent JDKs are used with CPUs that support AES-specific > instructions (https://en.wikipedia.org/wiki/AES_instruction_set). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
