I saw a similar 30-40% performance hit when testing a move from Plaintext to SASL_SSL.
It seemed to be due to the additional traffic generated by replication between brokers. Enabling SSL only between the client and brokers and leaving inter-broker traffic on Plaintext only introduced ~10% performance loss. Enabling SSL everywhere, but reducing either the number of replicas also saw a much smaller performance reduction. On Fri, Aug 24, 2018 at 9:05 AM Sri Harsha Chavali < sriharsha.chav...@outlook.com> wrote: > Hi Harsha, > > Given below are all the details. We are using Kafka On CDH. Do you have > any suggestion based on the below statistics. > > CDK - 2.2.0 - 0.10.2.0+kafka2.2.0+110 > Apache Kafka - 0.10.2. > java version "1.8.0_151" > > Tried Using Java 9 with not much difference. We need to make a small > change to kafka-run-class.sh file in order for it to pickup Java 9. > Statistics with Java 9, not very great. > 18/08/24 12:00:06 INFO utils.AppInfoParser: Kafka commitId : unknown > 28270 records sent, 5634.8 records/sec (6.45 MB/sec), 1892.8 ms avg > latency, 2988.0 max latency. > 60588 records sent, 12117.6 records/sec > (13.87 MB/sec), 2483.3 ms avg latency, 3410.0 max latency. > 59940 records sent, > 11973.6 records/sec (13.70 MB/sec), 2226.4 ms avg latency, 2436.0 max > latency. > 68364 records sent, 13648.2 records/sec (15.62 MB/sec), 1911.4 ms avg > latency, 2261.0 max latency. > 78570 records sent, 15714.0 records/sec (17.98 MB/sec), 1875.9 ms avg > latency, 2490.0 max latency. > 91368 records sent, 18259.0 records/sec (20.90 MB/sec), 1525.3 ms avg > latency, 1695.0 max latency. > 80838 records sent, 16141.8 records/sec (18.47 MB/sec), 1498.3 ms avg > latency, 2003.0 max latency. > 60750 records sent, 12140.3 records/sec (13.89 MB/sec), 2288.1 ms avg > latency, 2906.0 max latency. > 70146 records sent, 14015.2 records/sec (16.04 MB/sec), 2012.6 ms avg > latency, 2288.0 max latency. > 62208 records sent, 12439.1 records/sec (14.24 MB/sec), 2170.9 ms avg > latency, 2523.0 max latency. > 92502 records sent, 18493.0 records/sec (21.16 MB/sec), 1502.5 ms avg > latency, 2144.0 max latency. > 82458 records sent, 16491.6 records/sec (18.87 MB/sec), 1667.4 ms avg > latency, 1851.0 max latency. > 102708 records sent, 20537.5 records/sec (23.50 MB/sec), 1318.7 ms avg > latency, 1575.0 max latency. > 1000000 records sent, 14652.229337 records/sec (16.77 MB/sec), 1792.89 ms > avg latency, 3410.00 ms max latency, 1677 ms 50th, 2625 ms 95th, 3043 ms > 99th, 3372 ms 99.9th. > > > We use openssl to generate rsa:4096 bit keys. These are how the speeds > look like on the node. > > sign verify sign/s > verify/s > rsa 512 bits 0.000052s 0.000004s 19084.2 265528.2 > rsa 1024 bits 0.000194s 0.000010s 5160.4 96859.5 > rsa 2048 bits 0.001147s 0.000034s 872.1 29052.4 > rsa 4096 bits 0.008723s 0.000129s 114.6 7766.2 > > Thank you, > Harsha > Sent from Outlook<http://aka.ms/weboutlook> > ________________________________ > From: Harsha <ka...@harsha.io> > Sent: Thursday, August 23, 2018 3:42 PM > To: users@kafka.apache.org > Subject: Re: Performance Impact with Apache Kafka Security > > Hi, > Which Kafka version and Java version are you using? Did you try this > with Java 9 which has 2.5x perf improvements over Java 8 for SSL? Can you > try using a slightly weaker cipher suite to improve the performance? > > -Harsha > > On Wed, Aug 22, 2018, at 1:11 PM, Sri Harsha Chavali wrote: > > Hi Guys, > > > > We are trying to secure the Kafka-Cluster in order to enforce topic > > level security based on sentry roles. We are seeing a big performance > > impact after SSL_SASL is enabled. I read multiple blog posts describing > > the performance impact but that also said that the impact would be > > negligible, but I see a significant hit in my case (60-85%). Could > > someone please suggest if you have seen this kind of performance impact > > and what is done to overcome the same? We cannot afford to have an > > unsecure cluster. > > > > In the below example, I'm trying to produce a record of size 1200 bytes > > and set the batch.size property to 100000. > > > > Before Securityis Enabled: > > > > kafka-producer-perf-test --topic myPerformanceTestTopic --num-records > > 1000000 --print-metrics --record-size 1200 --throughput 1000000 -- > > producer-props acks=1 bootstrap.servers=<host_name>:9092 > > batch.size=100000 > > > > 18/08/22 10:12:37 INFO utils.AppInfoParser: Kafka commitId : unknown > > > > 294801 records sent, 58960.2 records/sec (67.47 MB/sec), 55.0 ms avg > > latency, 265.0 max latency. > > > > 275261 records sent, 49632.3 records/sec (56.80 MB/sec), 251.1 ms avg > > latency, 1420.0 max latency. > > > > 293654 records sent, 58730.8 records/sec (67.21 MB/sec), 244.0 ms avg > > latency, 1485.0 max latency. > > > > 1000000 records sent, 57733.387218 records/sec (66.07 MB/sec), 162.61 ms > > avg latency, 1485.00 ms max latency, 73 ms 50th, 546 ms 95th, 1459 ms > > 99th, 1477 ms 99.9th. > > > > After Security is Enabled: > > > > kafka-producer-perf-test --topic myPerformanceTestTopic --num-records > > 1000000 --print-metrics --record-size 1200 --throughput 1000000 -- > > producer-props acks=1 bootstrap.servers=<host_name>:9094 > > batch.size=100000 --producer.config <my_secure_path>/sasl-ssl- > > auth.properties > > > > > > 18/08/22 12:33:36 INFO utils.AppInfoParser: Kafka commitId : unknown > > > > 39610 records sent, 7917.2 records/sec (9.06 MB/sec), 1669.5 ms avg > > latency, 2608.0 max latency. > > > > 58320 records sent, 11659.3 records/sec (13.34 MB/sec), 2514.2 ms avg > > latency, 3242.0 max latency. > > > > 92016 records sent, 18399.5 records/sec (21.06 MB/sec), 1579.2 ms avg > > latency, 2119.0 max latency. > > > > 84645 records sent, 16925.6 records/sec (19.37 MB/sec), 1578.7 ms avg > > latency, 2111.0 max latency. > > > > 106515 records sent, 21286.0 records/sec (24.36 MB/sec), 1300.4 ms avg > > latency, 1662.0 max latency. > > > > l74520 records sent, 14895.1 records/sec (17.05 MB/sec), 1688.1 ms avg > > latency, 2350.0 max latency. > > > > 77841 records sent, 15562.0 records/sec (17.81 MB/sec), 1749.2 ms avg > > latency, 2030.0 max latency. > > > > 94851 records sent, 18970.2 records/sec (21.71 MB/sec), 1495.4 ms avg > > latency, 2111.0 max latency. > > > > 102870 records sent, 20569.9 records/sec (23.54 MB/sec), 1345.6 ms avg > > latency, 1559.0 max latency. > > > > 121095 records sent, 24219.0 records/sec (27.72 MB/sec), 1143.8 ms avg > > latency, 1738.0 max latency. > > > > 126036 records sent, 25202.2 records/sec (28.84 MB/sec), 1080.5 ms avg > > latency, 1384.0 max latency. > > > > 1000000 records sent, 17906.385417 records/sec (20.49 MB/sec), 1465.52 > > ms avg latency, 3242.00 ms max latency, 1355 ms 50th, 2339 ms 95th, 2914 > > ms 99th, 3211 ms 99.9th > > > > Thank you, > > Harsha >
