[jira] [Commented] (KAFKA-2561) Optionally support OpenSSL for SSL/TLS
[
https://issues.apache.org/jira/browse/KAFKA-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325333#comment-16325333
]
Prasanna Gautam commented on KAFKA-2561:
[~ijuma] is this still being planned anytime soon? I'd like to check if Java 9
or using Open/Boring/LibreSSL have any meaningful performance improvments for
SSL.
> Optionally support OpenSSL for SSL/TLS
> ---
>
> Key: KAFKA-2561
> URL: https://issues.apache.org/jira/browse/KAFKA-2561
> Project: Kafka
> Issue Type: New Feature
> Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>
> JDK's `SSLEngine` is unfortunately a bit slow (KAFKA-2431 covers this in more
> detail). We should consider supporting OpenSSL for SSL/TLS. Initial
> experiments on my laptop show that it performs a lot better:
> {code}
> start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg,
> nMsg.sec, config
> 2015-09-21 14:41:58:245, 2015-09-21 14:47:02:583, 28610.2295, 94.0081,
> 3000, 98574.6111, Java 8u60/server auth JDK
> SSLEngine/TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
> 2015-09-21 14:38:24:526, 2015-09-21 14:40:19:941, 28610.2295, 247.8900,
> 3000, 259931.5514, Java 8u60/server auth
> OpenSslEngine/TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
> 2015-09-21 14:49:03:062, 2015-09-21 14:50:27:764, 28610.2295, 337.7751,
> 3000, 354182.9000, Java 8u60/plaintext
> {code}
> Extracting the throughput figures:
> * JDK SSLEngine: 94 MB/s
> * OpenSSL SSLEngine: 247 MB/s
> * Plaintext: 337 MB/s (code from trunk, so no zero-copy due to KAFKA-2517)
> In order to get these figures, I used Netty's `OpenSslEngine` by hacking
> `SSLFactory` to use Netty's `SslContextBuilder` and made a few changes to
> `SSLTransportLayer` in order to workaround differences in behaviour between
> `OpenSslEngine` and JDK's SSLEngine (filed
> https://github.com/netty/netty/issues/4235 and
> https://github.com/netty/netty/issues/4238 upstream).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-6444) Kafka consumers and producers get confused by dualstack A + AAAA DNS records
Ivan Babrou created KAFKA-6444:
--
Summary: Kafka consumers and producers get confused by dualstack A
+ DNS records
Key: KAFKA-6444
URL: https://issues.apache.org/jira/browse/KAFKA-6444
Project: Kafka
Issue Type: Bug
Affects Versions: 0.11.0.1
Reporter: Ivan Babrou
We have hostnames with both A (IPv4) and (IPv6) DNS records. Kafka is
configured to only listen on IPv6 by manually setting IP to listen on and
advertise outside.
Brokers have no issue communicating between them, because they are not given
the option to resolve hostnames and pick IP protocol version.
Consumers and producers have to use bootstrap hostnames and do not try to
connect to IPv6 at all, they are stuck in SYN_SENT over IPv4:
{noformat}
syslog-ng 12621 999 123u IPv6 2411122889 0t0TCP
192.168.0.2:41626->192.168.0.1:9092 (SYN_SENT)
{noformat}
This happened to consumer in syslog-ng output plugin:
* https://github.com/balabit/syslog-ng/issues/1835
It also happened to a Flink consumer, although I do no have any more info about
that one. We fixed the issue by only providing records for bootstrapping.
Previously we saw the opposite problem with dualstack: software does not
implement happy eyeballs and only connects to IPv6 address, which is
firewalled. This happened to SSH (client gets stuck if you don't supply -4) and
Go (https://github.com/golang/go/issues/5) to give a couple of examples.
The solution for this is Happy Eyeballs:
https://en.wikipedia.org/wiki/Happy_Eyeballs
Kafka clients should connect to IPv6 first and then fallback to IPv4 if not
available.
There is also KAFKA-3347.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-3347) Configure java to prefer ipv4
[
https://issues.apache.org/jira/browse/KAFKA-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325293#comment-16325293
]
Ivan Babrou commented on KAFKA-3347:
Can this be closed as wontfix? Calendar says it's 2018 already.
> Configure java to prefer ipv4
> -
>
> Key: KAFKA-3347
> URL: https://issues.apache.org/jira/browse/KAFKA-3347
> Project: Kafka
> Issue Type: Improvement
> Components: core
>Affects Versions: 0.9.0.1
>Reporter: Jeremy Custenborder
>Assignee: Jeremy Custenborder
>Priority: Minor
>
> I've noticed that ports are sometimes binding on IPv6 addresses rather than
> the IPv4 address I'm expecting. Can we change this so we bing on the IPv4
> address rather than the IPv6 address? I'm proposing to add this to
> KAFKA_JVM_PERFORMANCE_OPTS.
> {code}
> -Djava.net.preferIPv4Stack=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6166) Streams configuration requires consumer. and producer. in order to be read
[
https://issues.apache.org/jira/browse/KAFKA-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325206#comment-16325206
]
Filipe Agapito commented on KAFKA-6166:
---
This looks like a nice newbie issue. I'm going to assign it to myself and work
on it.
> Streams configuration requires consumer. and producer. in order to be read
> --
>
> Key: KAFKA-6166
> URL: https://issues.apache.org/jira/browse/KAFKA-6166
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
> Environment: Kafka 0.11.0.0
> JDK 1.8
> CoreOS
>Reporter: Justin Manchester
>Priority: Minor
> Labels: newbie++, user-experience
>
> Problem:
> In previous release you could specify a custom metrics reporter like so:
> Properties config = new Properties();
> config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker1:9092");
> config.put(StreamsConfig.METRIC_REPORTER_CLASSES_CONFIG,
> "com.mycompany.MetricReporter");
> config.put("custom-key-for-metric-reporter", "value");
> From 0.11.0.0 onwards this is no longer possible, as you have to specify
> consumer.custom-key-for-metric-reporter or
> producer.custom-key-for-metric-reporter otherwise it's stripped out of the
> configuration.
> So, if you wish to use a metrics reporter and to collect producer and
> consumer metrics, as well as kafka-streams metrics, that you would need to
> specify 3 distinct configs:
> 1) consumer.custom-key-for-metric-reporter
> 2) producer.custom-key-for-metric-reporter
> 3) custom-key-for-metric-reporter
> This appears to be a regression.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (KAFKA-6166) Streams configuration requires consumer. and producer. in order to be read
[
https://issues.apache.org/jira/browse/KAFKA-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Filipe Agapito reassigned KAFKA-6166:
-
Assignee: Filipe Agapito
> Streams configuration requires consumer. and producer. in order to be read
> --
>
> Key: KAFKA-6166
> URL: https://issues.apache.org/jira/browse/KAFKA-6166
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
> Environment: Kafka 0.11.0.0
> JDK 1.8
> CoreOS
>Reporter: Justin Manchester
>Assignee: Filipe Agapito
>Priority: Minor
> Labels: newbie++, user-experience
>
> Problem:
> In previous release you could specify a custom metrics reporter like so:
> Properties config = new Properties();
> config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker1:9092");
> config.put(StreamsConfig.METRIC_REPORTER_CLASSES_CONFIG,
> "com.mycompany.MetricReporter");
> config.put("custom-key-for-metric-reporter", "value");
> From 0.11.0.0 onwards this is no longer possible, as you have to specify
> consumer.custom-key-for-metric-reporter or
> producer.custom-key-for-metric-reporter otherwise it's stripped out of the
> configuration.
> So, if you wish to use a metrics reporter and to collect producer and
> consumer metrics, as well as kafka-streams metrics, that you would need to
> specify 3 distinct configs:
> 1) consumer.custom-key-for-metric-reporter
> 2) producer.custom-key-for-metric-reporter
> 3) custom-key-for-metric-reporter
> This appears to be a regression.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6409) LogRecoveryTest (testHWCheckpointWithFailuresSingleLogSegment) is flaky
[ https://issues.apache.org/jira/browse/KAFKA-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wladimir Schmidt updated KAFKA-6409: Description: In the LogRecoveryTest the test named testHWCheckpointWithFailuresSingleLogSegment is affected and not stable. Sometimes it passes, sometimes it is not. Scala 2.12. JDK9 java.lang.AssertionError: Timing out after 3 ms since a new leader that is different from 1 was not elected for partition new-topic-0, leader is Some(1) at kafka.utils.TestUtils$.fail(TestUtils.scala:351) at kafka.utils.TestUtils$.$anonfun$waitUntilLeaderIsElectedOrChanged$8(TestUtils.scala:828) at scala.Option.getOrElse(Option.scala:121) at kafka.utils.TestUtils$.waitUntilLeaderIsElectedOrChanged(TestUtils.scala:818) at kafka.server.LogRecoveryTest.testHWCheckpointWithFailuresSingleLogSegment(LogRecoveryTest.scala:152) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) at com.sun.proxy.$Proxy1.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:108) at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:146) at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:128) at
