[jira] [Commented] (KAFKA-7009) Mute logger for reflections.org at the warn level in system tests
[
https://issues.apache.org/jira/browse/KAFKA-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510660#comment-16510660
]
ASF GitHub Bot commented on KAFKA-7009:
---
ewencp closed pull request #5151: KAFKA-7009: Suppress the Reflections log
warning messages in system tests
URL: https://github.com/apache/kafka/pull/5151
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/tests/kafkatest/services/templates/connect_log4j.properties
b/tests/kafkatest/services/templates/connect_log4j.properties
index c972b1df86c..adb35b30bfc 100644
--- a/tests/kafkatest/services/templates/connect_log4j.properties
+++ b/tests/kafkatest/services/templates/connect_log4j.properties
@@ -27,3 +27,4 @@ log4j.appender.FILE.layout.conversionPattern=[%d] %p %m (%c)%n
log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
+log4j.logger.org.reflections=ERROR
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Mute logger for reflections.org at the warn level in system tests
> -
>
> Key: KAFKA-7009
> URL: https://issues.apache.org/jira/browse/KAFKA-7009
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect, system tests
>Affects Versions: 1.1.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Critical
> Fix For: 0.10.0.2, 0.10.1.2, 0.10.2.2, 2.0.0, 0.11.0.3, 2.1.0
>
>
> AK's Log4J configuration file for Connect includes [these
> lines|https://github.com/apache/kafka/blob/trunk/config/connect-log4j.properties#L25]:
> {code}
> log4j.logger.org.apache.zookeeper=ERROR
> log4j.logger.org.I0Itec.zkclient=ERROR
> log4j.logger.org.reflections=ERROR
> {code}
> The last one suppresses lots of Reflections warnings like the following that
> are output during classpath scanning and are harmless:
> {noformat}
> [2018-06-06 13:52:39,448] WARN could not create Vfs.Dir from url. ignoring
> the exception and continuing (org.reflections.Reflections)
> org.reflections.ReflectionsException: could not create Vfs.Dir from url, no
> matching UrlType was found
> [file:/usr/bin/../share/java/confluent-support-metrics/*]
> either use fromURL(final URL url, final List urlTypes) or use the
> static setDefaultURLTypes(final List urlTypes) or
> addDefaultURLTypes(UrlType urlType) with your specialized UrlType.
> at org.reflections.vfs.Vfs.fromURL(Vfs.java:109)
> at org.reflections.vfs.Vfs.fromURL(Vfs.java:91)
> at org.reflections.Reflections.scan(Reflections.java:240)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.scan(DelegatingClassLoader.java:373)
> at org.reflections.Reflections$1.run(Reflections.java:198)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The last line also need to be added to [Connect's Log4J configuration file in
> the AK system
> tests|https://github.com/apache/kafka/blob/trunk/tests/kafkatest/services/templates/connect_log4j.properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-7009) Mute logger for reflections.org at the warn level in system tests
[
https://issues.apache.org/jira/browse/KAFKA-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewen Cheslack-Postava resolved KAFKA-7009.
--
Resolution: Fixed
Fix Version/s: 2.1.0
0.11.0.3
0.10.2.2
0.10.1.2
0.10.0.2
Issue resolved by pull request 5151
[https://github.com/apache/kafka/pull/5151]
> Mute logger for reflections.org at the warn level in system tests
> -
>
> Key: KAFKA-7009
> URL: https://issues.apache.org/jira/browse/KAFKA-7009
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect, system tests
>Affects Versions: 1.1.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Critical
> Fix For: 0.10.0.2, 0.10.1.2, 0.10.2.2, 2.0.0, 0.11.0.3, 2.1.0
>
>
> AK's Log4J configuration file for Connect includes [these
> lines|https://github.com/apache/kafka/blob/trunk/config/connect-log4j.properties#L25]:
> {code}
> log4j.logger.org.apache.zookeeper=ERROR
> log4j.logger.org.I0Itec.zkclient=ERROR
> log4j.logger.org.reflections=ERROR
> {code}
> The last one suppresses lots of Reflections warnings like the following that
> are output during classpath scanning and are harmless:
> {noformat}
> [2018-06-06 13:52:39,448] WARN could not create Vfs.Dir from url. ignoring
> the exception and continuing (org.reflections.Reflections)
> org.reflections.ReflectionsException: could not create Vfs.Dir from url, no
> matching UrlType was found
> [file:/usr/bin/../share/java/confluent-support-metrics/*]
> either use fromURL(final URL url, final List urlTypes) or use the
> static setDefaultURLTypes(final List urlTypes) or
> addDefaultURLTypes(UrlType urlType) with your specialized UrlType.
> at org.reflections.vfs.Vfs.fromURL(Vfs.java:109)
> at org.reflections.vfs.Vfs.fromURL(Vfs.java:91)
> at org.reflections.Reflections.scan(Reflections.java:240)
> at
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.scan(DelegatingClassLoader.java:373)
> at org.reflections.Reflections$1.run(Reflections.java:198)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The last line also need to be added to [Connect's Log4J configuration file in
> the AK system
> tests|https://github.com/apache/kafka/blob/trunk/tests/kafkatest/services/templates/connect_log4j.properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6161) Introduce new serdes interfaces with empty configure() and close()
[
https://issues.apache.org/jira/browse/KAFKA-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510653#comment-16510653
]
Chia-Ping Tsai commented on KAFKA-6161:
---
[~ewencp] Got it. thanks!
> Introduce new serdes interfaces with empty configure() and close()
> --
>
> Key: KAFKA-6161
> URL: https://issues.apache.org/jira/browse/KAFKA-6161
> Project: Kafka
> Issue Type: Improvement
> Components: clients, streams
>Reporter: Evgeny Veretennikov
>Assignee: Evgeny Veretennikov
>Priority: Major
> Labels: needs-kip
>
> {{Serializer}}, {{Deserializer}} and {{Serde}} interfaces have methods
> {{configure()}} and {{close()}}. Pretty often one want to leave these methods
> empty. For example, a lot of serializers inside
> {{org.apache.kafka.common.serialization}} package have these methods empty:
> {code}
> @Override
> public void configure(Map configs, boolean isKey) {
> // nothing to do
> }
> @Override
> public void close() {
> // nothing to do
> }
> {code}
> To avoid such boilerplate, we may create new interfaces (like
> {{UnconfiguredSerializer}}), in which we will define these methods empty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7031) Kafka Connect API module depends on Jersey
[
https://issues.apache.org/jira/browse/KAFKA-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510631#comment-16510631
]
ASF GitHub Bot commented on KAFKA-7031:
---
ewencp closed pull request #5190: KAFKA-7031 : Connect API shouldn't depend on
jersey
URL: https://github.com/apache/kafka/pull/5190
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/build.gradle b/build.gradle
index b796c218e21..7db8ddcf6d1 100644
--- a/build.gradle
+++ b/build.gradle
@@ -1210,7 +1210,7 @@ project(':connect:api') {
dependencies {
compile project(':clients')
compile libs.slf4jApi
-compile libs.jerseyContainerServlet
+compile libs.jaxrsApi
testCompile libs.junit
@@ -1450,6 +1450,7 @@ project(':connect:basic-auth-extension') {
testCompile project(':clients').sourceSets.test.output
testRuntime libs.slf4jlog4j
+testRuntime libs.jerseyContainerServlet
}
javadoc {
diff --git a/gradle/dependencies.gradle b/gradle/dependencies.gradle
index ce2db721025..c9f6f0349a5 100644
--- a/gradle/dependencies.gradle
+++ b/gradle/dependencies.gradle
@@ -60,6 +60,7 @@ versions += [
log4j: "1.2.17",
scalaLogging: "3.9.0",
jaxb: "2.3.0",
+ jaxrs: "2.1",
jfreechart: "1.0.0",
jopt: "5.0.4",
junit: "4.12",
@@ -101,6 +102,7 @@ libs += [
jacksonDatabind:
"com.fasterxml.jackson.core:jackson-databind:$versions.jackson",
jacksonJaxrsJsonProvider:
"com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:$versions.jackson",
jaxbApi: "javax.xml.bind:jaxb-api:$versions.jaxb",
+ jaxrsApi: "javax.ws.rs:javax.ws.rs-api:$versions.jaxrs",
jettyServer: "org.eclipse.jetty:jetty-server:$versions.jetty",
jettyClient: "org.eclipse.jetty:jetty-client:$versions.jetty",
jettyServlet: "org.eclipse.jetty:jetty-servlet:$versions.jetty",
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Kafka Connect API module depends on Jersey
> --
>
> Key: KAFKA-7031
> URL: https://issues.apache.org/jira/browse/KAFKA-7031
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.0
>Reporter: Randall Hauch
>Assignee: Magesh kumar Nandakumar
>Priority: Blocker
> Fix For: 2.0.0, 2.1.0
>
>
> The Kafka Connect API module for 2.0.0 brings in Jersey dependencies. When I
> run {{mvn dependency:tree}} on a project that depends only on the snapshot
> version of {{org.apache.kafka:kafka-connect-api}}, the following are shown:
> {noformat}
> [INFO] +- org.apache.kafka:connect-api:jar:2.0.0-SNAPSHOT:compile
> [INFO] | +- org.slf4j:slf4j-api:jar:1.7.25:compile
> [INFO] | \-
> org.glassfish.jersey.containers:jersey-container-servlet:jar:2.27:compile
> [INFO] | +-
> org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.27:compile
> [INFO] | | \-
> org.glassfish.hk2.external:javax.inject:jar:2.5.0-b42:compile
> [INFO] | +- org.glassfish.jersey.core:jersey-common:jar:2.27:compile
> [INFO] | | +- javax.annotation:javax.annotation-api:jar:1.2:compile
> [INFO] | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
> [INFO] | +- org.glassfish.jersey.core:jersey-server:jar:2.27:compile
> [INFO] | | +- org.glassfish.jersey.core:jersey-client:jar:2.27:compile
> [INFO] | | +-
> org.glassfish.jersey.media:jersey-media-jaxb:jar:2.27:compile
> [INFO] | | \- javax.validation:validation-api:jar:1.1.0.Final:compile
> [INFO] | \- javax.ws.rs:javax.ws.rs-api:jar:2.1:compile
> ...
> {noformat}
> This may have been an unintended side effect of the
> [KIP-285|https://cwiki.apache.org/confluence/display/KAFKA/KIP-285%3A+Connect+Rest+Extension+Plugin]
> effort, which added the REST extension for Connect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-7031) Kafka Connect API module depends on Jersey
[
https://issues.apache.org/jira/browse/KAFKA-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewen Cheslack-Postava resolved KAFKA-7031.
--
Resolution: Fixed
Fix Version/s: 2.1.0
Issue resolved by pull request 5190
[https://github.com/apache/kafka/pull/5190]
> Kafka Connect API module depends on Jersey
> --
>
> Key: KAFKA-7031
> URL: https://issues.apache.org/jira/browse/KAFKA-7031
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.0
>Reporter: Randall Hauch
>Assignee: Magesh kumar Nandakumar
>Priority: Blocker
> Fix For: 2.0.0, 2.1.0
>
>
> The Kafka Connect API module for 2.0.0 brings in Jersey dependencies. When I
> run {{mvn dependency:tree}} on a project that depends only on the snapshot
> version of {{org.apache.kafka:kafka-connect-api}}, the following are shown:
> {noformat}
> [INFO] +- org.apache.kafka:connect-api:jar:2.0.0-SNAPSHOT:compile
> [INFO] | +- org.slf4j:slf4j-api:jar:1.7.25:compile
> [INFO] | \-
> org.glassfish.jersey.containers:jersey-container-servlet:jar:2.27:compile
> [INFO] | +-
> org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.27:compile
> [INFO] | | \-
> org.glassfish.hk2.external:javax.inject:jar:2.5.0-b42:compile
> [INFO] | +- org.glassfish.jersey.core:jersey-common:jar:2.27:compile
> [INFO] | | +- javax.annotation:javax.annotation-api:jar:1.2:compile
> [INFO] | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
> [INFO] | +- org.glassfish.jersey.core:jersey-server:jar:2.27:compile
> [INFO] | | +- org.glassfish.jersey.core:jersey-client:jar:2.27:compile
> [INFO] | | +-
> org.glassfish.jersey.media:jersey-media-jaxb:jar:2.27:compile
> [INFO] | | \- javax.validation:validation-api:jar:1.1.0.Final:compile
> [INFO] | \- javax.ws.rs:javax.ws.rs-api:jar:2.1:compile
> ...
> {noformat}
> This may have been an unintended side effect of the
> [KIP-285|https://cwiki.apache.org/confluence/display/KAFKA/KIP-285%3A+Connect+Rest+Extension+Plugin]
> effort, which added the REST extension for Connect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6161) Introduce new serdes interfaces with empty configure() and close()
[
https://issues.apache.org/jira/browse/KAFKA-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510622#comment-16510622
]
Ewen Cheslack-Postava commented on KAFKA-6161:
--
[~chia7712] Really up to [~evis] (we prefer to have existing assignee pass
ownership on), but if we don't hear back, then sure, feel free to re-assign to
yourself.
> Introduce new serdes interfaces with empty configure() and close()
> --
>
> Key: KAFKA-6161
> URL: https://issues.apache.org/jira/browse/KAFKA-6161
> Project: Kafka
> Issue Type: Improvement
> Components: clients, streams
>Reporter: Evgeny Veretennikov
>Assignee: Evgeny Veretennikov
>Priority: Major
> Labels: needs-kip
>
> {{Serializer}}, {{Deserializer}} and {{Serde}} interfaces have methods
> {{configure()}} and {{close()}}. Pretty often one want to leave these methods
> empty. For example, a lot of serializers inside
> {{org.apache.kafka.common.serialization}} package have these methods empty:
> {code}
> @Override
> public void configure(Map configs, boolean isKey) {
> // nothing to do
> }
> @Override
> public void close() {
> // nothing to do
> }
> {code}
> To avoid such boilerplate, we may create new interfaces (like
> {{UnconfiguredSerializer}}), in which we will define these methods empty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6914) Kafka Connect - Plugins class should have a constructor that can take in parent ClassLoader
[ https://issues.apache.org/jira/browse/KAFKA-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510619#comment-16510619 ] Ewen Cheslack-Postava commented on KAFKA-6914: -- [~sriramks85] Spring Boot is handy for creating your own apps, but I'm not sure it's really the right solution for *repackaging* an existing app. Spring Boot combined with pluggable systems that require plugin discovery are likely to generally run into problems. Since it's a custom approach to Spring Boot, it's hard to generically support that deployment model. The current solution you have is definitely touching internal APIs that are not guaranteed to be stable, so it's not a great long term solution. The proposed fix is also relying on internal APIs that, again, are not guaranteed to be stable. I think a real solution here probably requires some further thought, more broadly than just Spring Boot, about how custom classloaders are used and how they interact with plugin-based systems like Connect. Are there any similar examples that you know of for plugin-based systems that face these same challenges? And relatedly, for your particular case, does just using an uber jar approach not work? Are you trying to support a variety of connectors with potentially conflicting dependencies, or are you packaging a single connector for deployment on OpenShift? [~flyaruu] This seems like it is probably reasonable – I don't think you're missing intention, but rather from the 99% use case we see the system classloader *is* the DelegatingClassLoader's classloader, so it's probably more of an oversight/bug. [~kkonstantine] thoughts? Seems like a pretty trivial fix (though difficult to include good tests for, as per usual with this classloader stuff). > Kafka Connect - Plugins class should have a constructor that can take in > parent ClassLoader > --- > > Key: KAFKA-6914 > URL: https://issues.apache.org/jira/browse/KAFKA-6914 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Sriram KS >Priority: Minor > > Currently Plugins class has a single constructor that takes in map of props. > Please make Plugin class to have a constructor that takes in a classLoader as > well and use it to set DelegationClassLoader's parent classLoader. > Reason: > This will be useful if i am already having a managed class Loader environment > like a Spring boot app which resolves my class dependencies using my > maven/gradle dependency management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6914) Kafka Connect - Plugins class should have a constructor that can take in parent ClassLoader
[ https://issues.apache.org/jira/browse/KAFKA-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-6914: - Fix Version/s: (was: 1.1.1) > Kafka Connect - Plugins class should have a constructor that can take in > parent ClassLoader > --- > > Key: KAFKA-6914 > URL: https://issues.apache.org/jira/browse/KAFKA-6914 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Sriram KS >Priority: Minor > > Currently Plugins class has a single constructor that takes in map of props. > Please make Plugin class to have a constructor that takes in a classLoader as > well and use it to set DelegationClassLoader's parent classLoader. > Reason: > This will be useful if i am already having a managed class Loader environment > like a Spring boot app which resolves my class dependencies using my > maven/gradle dependency management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7043) Connect isolation whitelist does not include new primitive converters (KIP-305)
[
https://issues.apache.org/jira/browse/KAFKA-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510609#comment-16510609
]
ASF GitHub Bot commented on KAFKA-7043:
---
ewencp closed pull request #5198: KAFKA-7043: Modified plugin isolation
whitelist with recently added converters (KIP-305)
URL: https://github.com/apache/kafka/pull/5198
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java
b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java
index b4aee4741c3..7ca810b3dec 100644
---
a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java
+++
b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java
@@ -127,6 +127,11 @@
+ "|json\\..*"
+ "|file\\..*"
+ "|converters\\..*"
++ "|storage\\.DoubleConverter"
++ "|storage\\.FloatConverter"
++ "|storage\\.IntegerConverter"
++ "|storage\\.LongConverter"
++ "|storage\\.ShortConverter"
+ "|storage\\.StringConverter"
+
"|rest\\.basic\\.auth\\.extension\\.BasicAuthSecurityRestExtension"
+ "))$";
diff --git
a/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/isolation/PluginUtilsTest.java
b/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/isolation/PluginUtilsTest.java
index 9698153f986..45aea975ef2 100644
---
a/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/isolation/PluginUtilsTest.java
+++
b/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/isolation/PluginUtilsTest.java
@@ -143,6 +143,21 @@ public void testAllowedConnectFrameworkClasses() throws
Exception {
assertTrue(PluginUtils.shouldLoadInIsolation(
"org.apache.kafka.connect.converters.ByteArrayConverter")
);
+assertTrue(PluginUtils.shouldLoadInIsolation(
+"org.apache.kafka.connect.storage.DoubleConverter")
+);
+assertTrue(PluginUtils.shouldLoadInIsolation(
+"org.apache.kafka.connect.storage.FloatConverter")
+);
+assertTrue(PluginUtils.shouldLoadInIsolation(
+"org.apache.kafka.connect.storage.IntegerConverter")
+);
+assertTrue(PluginUtils.shouldLoadInIsolation(
+"org.apache.kafka.connect.storage.LongConverter")
+);
+assertTrue(PluginUtils.shouldLoadInIsolation(
+"org.apache.kafka.connect.storage.ShortConverter")
+);
assertTrue(PluginUtils.shouldLoadInIsolation(
"org.apache.kafka.connect.storage.StringConverter")
);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Connect isolation whitelist does not include new primitive converters
> (KIP-305)
> ---
>
> Key: KAFKA-7043
> URL: https://issues.apache.org/jira/browse/KAFKA-7043
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Blocker
> Fix For: 2.0.0, 2.1.0
>
>
> KIP-305 added several new primitive converters, but the PR did not add them
> to the whitelist for the plugin isolation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-7043) Connect isolation whitelist does not include new primitive converters (KIP-305)
[ https://issues.apache.org/jira/browse/KAFKA-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-7043. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 5198 [https://github.com/apache/kafka/pull/5198] > Connect isolation whitelist does not include new primitive converters > (KIP-305) > --- > > Key: KAFKA-7043 > URL: https://issues.apache.org/jira/browse/KAFKA-7043 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.0.0 >Reporter: Randall Hauch >Assignee: Randall Hauch >Priority: Blocker > Fix For: 2.0.0, 2.1.0 > > > KIP-305 added several new primitive converters, but the PR did not add them > to the whitelist for the plugin isolation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6906) Kafka Streams does not commit transactions if data is produced via wall-clock punctuation
[ https://issues.apache.org/jira/browse/KAFKA-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-6906: --- Fix Version/s: 1.0.2 0.11.0.3 > Kafka Streams does not commit transactions if data is produced via wall-clock > punctuation > - > > Key: KAFKA-6906 > URL: https://issues.apache.org/jira/browse/KAFKA-6906 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 1.1.0 >Reporter: Matthias J. Sax >Assignee: Jagadesh Adireddi >Priority: Major > Fix For: 2.0.0, 0.11.0.3, 1.0.2, 1.1.1 > > > Committing in Kafka Streams happens in regular intervals. However, committing > only happens if new input records got processed since the last commit (via > setting flag `commitOffsetNeeded` within `StreamTask#process()`) > However, data could also be emitted via wall-clock based punctuation calls. > Especially if EOS is enabled, this is an issue (maybe also for non-EOS) > because the current running transaction is not committed and thus might time > out leading to a fatal error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6634) Delay initiating the txn on producers until initializeTopology with EOS turned on
[ https://issues.apache.org/jira/browse/KAFKA-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-6634: --- Fix Version/s: 1.0.2 0.11.0.3 > Delay initiating the txn on producers until initializeTopology with EOS > turned on > - > > Key: KAFKA-6634 > URL: https://issues.apache.org/jira/browse/KAFKA-6634 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Guozhang Wang >Assignee: Guozhang Wang >Priority: Major > Fix For: 1.1.0, 0.11.0.3, 1.0.2 > > > In Streams EOS implementation, the created producers for tasks will initiate > a txn immediately after being created in the constructor of `StreamTask`. > However, the task may not process any data and hence producer may not send > any records for that started txn for a long time because of the restoration > process. And with default txn.session.timeout valued at 60 seconds, it means > that if the restoration takes more than that amount of time, upon starting > the producer will immediately get the error that its producer epoch is > already old. > To fix this, we should consider instantiating the txn only after the > restoration phase is done. Although this may have a caveat that if the > producer is already fenced, it will not be notified until then, in > initializeTopology. But I think this should not be a correctness issue since > during the restoration process we do not make any changes to the processing > state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7051) Improve the efficiency of the ReplicaManager when there are many partitions
[ https://issues.apache.org/jira/browse/KAFKA-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510469#comment-16510469 ] ASF GitHub Bot commented on KAFKA-7051: --- guozhangwang opened a new pull request #5207: KAFKA-7051: checkpoint offsets from committed URL: https://github.com/apache/kafka/pull/5207 This is a cherry-pick PR of KAFKA-7051 for 1.1 and older. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve the efficiency of the ReplicaManager when there are many partitions > --- > > Key: KAFKA-7051 > URL: https://issues.apache.org/jira/browse/KAFKA-7051 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6205) Have State Stores Restore Before Initializing Toplogy
[ https://issues.apache.org/jira/browse/KAFKA-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-6205: --- Fix Version/s: 0.11.0.3 > Have State Stores Restore Before Initializing Toplogy > - > > Key: KAFKA-6205 > URL: https://issues.apache.org/jira/browse/KAFKA-6205 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 1.0.0, 0.11.0.2 >Reporter: Bill Bejeck >Assignee: Bill Bejeck >Priority: Major > Fix For: 1.1.0, 1.0.1, 0.11.0.3 > > > Streams should restore state stores (if needed) before initializing the > topology. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7049) InternalTopicIntegrationTest sometimes fails
[
https://issues.apache.org/jira/browse/KAFKA-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7049:
---
Component/s: unit tests
streams
> InternalTopicIntegrationTest sometimes fails
>
>
> Key: KAFKA-7049
> URL: https://issues.apache.org/jira/browse/KAFKA-7049
> Project: Kafka
> Issue Type: Test
> Components: streams, unit tests
>Reporter: Ted Yu
>Priority: Minor
>
> Saw the following based on commit fa1d0383902260576132e09bdf9efcc2784b55b4 :
> {code}
> org.apache.kafka.streams.integration.InternalTopicIntegrationTest >
> shouldCompactTopicsForKeyValueStoreChangelogs FAILED
> java.lang.RuntimeException: Timed out waiting for completion.
> lagMetrics=[0/2] totalLag=[0.0]
> at
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForCompletion(IntegrationTestUtils.java:227)
> at
> org.apache.kafka.streams.integration.InternalTopicIntegrationTest.shouldCompactTopicsForKeyValueStoreChangelogs(InternalTopicIntegrationTest.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7044) kafka-consumer-groups.sh NullPointerException describing round robin or sticky assignors
[
https://issues.apache.org/jira/browse/KAFKA-7044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510405#comment-16510405
]
Jeff Field commented on KAFKA-7044:
---
{code:java}
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group
groupname{code}
Is all I need to do to reproduce it. We've got 18 consumers spread across 6
hosts (3 per host) in the consumer group reading from 2 topics with 12
partitions and 2 topics with 18 partitions (60 total) spread across 3 brokers.
All partitions have received and are receiving data. Currently we're using
burrow to get assignment information since we can't get it out of the
kafka-consumer-groups command, I can post the burrow output for the group if
that is helpful (but nothing stands out to me)
> kafka-consumer-groups.sh NullPointerException describing round robin or
> sticky assignors
>
>
> Key: KAFKA-7044
> URL: https://issues.apache.org/jira/browse/KAFKA-7044
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Affects Versions: 1.1.0
> Environment: CentOS 7.4, Oracle JDK 1.8
>Reporter: Jeff Field
>Assignee: Vahid Hashemian
>Priority: Minor
>
> We've recently moved to using the round robin assignor for one of our
> consumer groups, and started testing the sticky assignor. In both cases,
> using Kafka 1.1.0 we get a null pointer exception *unless* the group being
> described is rebalancing:
> {code:java}
> kafka-consumer-groups --bootstrap-server fqdn:9092 --describe --group
> groupname-for-consumer
> Error: Executing consumer group command failed due to null
> [2018-06-12 01:32:34,179] DEBUG Exception in consumer group command
> (kafka.admin.ConsumerGroupCommand$)
> java.lang.NullPointerException
> at scala.Predef$.Long2long(Predef.scala:363)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$getLogEndOffsets$2.apply(ConsumerGroupCommand.scala:612)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$getLogEndOffsets$2.apply(ConsumerGroupCommand.scala:610)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.immutable.List.map(List.scala:296)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.getLogEndOffsets(ConsumerGroupCommand.scala:610)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describePartitions(ConsumerGroupCommand.scala:328)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.collectConsumerAssignment(ConsumerGroupCommand.scala:308)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.collectConsumerAssignment(ConsumerGroupCommand.scala:544)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10$$anonfun$13.apply(ConsumerGroupCommand.scala:571)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10$$anonfun$13.apply(ConsumerGroupCommand.scala:565)
> at scala.collection.immutable.List.flatMap(List.scala:338)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10.apply(ConsumerGroupCommand.scala:565)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10.apply(ConsumerGroupCommand.scala:558)
> at scala.Option.map(Option.scala:146)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.collectGroupOffsets(ConsumerGroupCommand.scala:558)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describeGroup(ConsumerGroupCommand.scala:271)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:544)
> at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:77)
> at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
> [2018-06-12 01:32:34,255] DEBUG Removed sensor with name connections-closed:
> (org.apache.kafka.common.metrics.Metrics){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7051) Improve the efficiency of the ReplicaManager when there are many partitions
[ https://issues.apache.org/jira/browse/KAFKA-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-7051: Fix Version/s: 2.0.0 > Improve the efficiency of the ReplicaManager when there are many partitions > --- > > Key: KAFKA-7051 > URL: https://issues.apache.org/jira/browse/KAFKA-7051 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6979) Add max.block.ms to consumer for default timeout behavior
[
https://issues.apache.org/jira/browse/KAFKA-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510380#comment-16510380
]
ASF GitHub Bot commented on KAFKA-6979:
---
hachikuji closed pull request #5122: KAFKA-6979: Add default.api.timeout.ms to
KafkaConsumer (KIP-266)
URL: https://github.com/apache/kafka/pull/5122
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java
b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java
index 72e496cbd46..bc9a716158e 100644
---
a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java
+++
b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java
@@ -218,6 +218,10 @@
public static final String REQUEST_TIMEOUT_MS_CONFIG =
CommonClientConfigs.REQUEST_TIMEOUT_MS_CONFIG;
private static final String REQUEST_TIMEOUT_MS_DOC =
CommonClientConfigs.REQUEST_TIMEOUT_MS_DOC;
+/** default.api.timeout.ms */
+public static final String DEFAULT_API_TIMEOUT_MS_CONFIG =
"default.api.timeout.ms";
+public static final String DEFAULT_API_TIMEOUT_MS_DOC = "Specifies the
timeout (in milliseconds) for consumer APIs that could block. This
configuration is used as the default timeout for all consumer operations that
do not explicitly accept a timeout parameter.";
+
/** interceptor.classes */
public static final String INTERCEPTOR_CLASSES_CONFIG =
"interceptor.classes";
public static final String INTERCEPTOR_CLASSES_DOC = "A list of classes to
use as interceptors. "
@@ -403,6 +407,12 @@
atLeast(0),
Importance.MEDIUM,
REQUEST_TIMEOUT_MS_DOC)
+.define(DEFAULT_API_TIMEOUT_MS_CONFIG,
+Type.INT,
+60 * 1000,
+atLeast(0),
+Importance.MEDIUM,
+DEFAULT_API_TIMEOUT_MS_DOC)
/* default is set to be a bit lower than the
server default (10 min), to avoid both client and server closing connection at
same time */
.define(CONNECTIONS_MAX_IDLE_MS_CONFIG,
Type.LONG,
diff --git
a/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
b/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
index 5bd6b935b39..d6973c0a818 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
@@ -567,6 +567,7 @@
private final Metadata metadata;
private final long retryBackoffMs;
private final long requestTimeoutMs;
+private final int defaultApiTimeoutMs;
private volatile boolean closed = false;
private List assignors;
@@ -666,6 +667,7 @@ private KafkaConsumer(ConsumerConfig config,
log.debug("Initializing the Kafka consumer");
this.requestTimeoutMs =
config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
+this.defaultApiTimeoutMs =
config.getInt(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG);
int sessionTimeOutMs =
config.getInt(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG);
int fetchMaxWaitMs =
config.getInt(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG);
if (this.requestTimeoutMs <= sessionTimeOutMs ||
this.requestTimeoutMs <= fetchMaxWaitMs)
@@ -814,6 +816,7 @@ private KafkaConsumer(ConsumerConfig config,
Metadata metadata,
long retryBackoffMs,
long requestTimeoutMs,
+ int defaultApiTimeoutMs,
List assignors) {
this.log = logContext.logger(getClass());
this.clientId = clientId;
@@ -829,6 +832,7 @@ private KafkaConsumer(ConsumerConfig config,
this.metadata = metadata;
this.retryBackoffMs = retryBackoffMs;
this.requestTimeoutMs = requestTimeoutMs;
+this.defaultApiTimeoutMs = defaultApiTimeoutMs;
this.assignors = assignors;
}
@@ -1268,8 +1272,9 @@ private long remainingTimeAtLeastZero(final long
timeoutMs, final long elapsedTi
* every rebalance and also on startup. As such, if you need to store
offsets in anything other than Kafka, this API
* should not be used.
*
- * This is a synchronous commits and will block until either the commit
succeeds or an
[jira] [Commented] (KAFKA-7051) Improve the efficiency of the ReplicaManager when there are many partitions
[ https://issues.apache.org/jira/browse/KAFKA-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510359#comment-16510359 ] ASF GitHub Bot commented on KAFKA-7051: --- cmccabe opened a new pull request #5206: KAFKA-7051: Improve the efficiency of ReplicaManager URL: https://github.com/apache/kafka/pull/5206 Improve the efficiency of the ReplicaManager when there are many partitions. The 'fetchInfo' argument to ReplicaManager#fetchMessages can be an iterator rather than a sequence. This avoids the need to materialize a new collection in some cases. Reduce the number of times we loop over the read results obtained from 'readFromLog'. Fix a case where we were performing an O(N^2) operation to join the set of partitions we wanted info about, and the set of partitions we obtained information about. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve the efficiency of the ReplicaManager when there are many partitions > --- > > Key: KAFKA-7051 > URL: https://issues.apache.org/jira/browse/KAFKA-7051 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.0 >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7051) Improve the efficiency of the ReplicaManager when there are many partitions
Colin P. McCabe created KAFKA-7051: -- Summary: Improve the efficiency of the ReplicaManager when there are many partitions Key: KAFKA-7051 URL: https://issues.apache.org/jira/browse/KAFKA-7051 Project: Kafka Issue Type: Bug Components: replication Affects Versions: 0.8.0 Reporter: Colin P. McCabe Assignee: Colin P. McCabe -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7010) Rename ResourceNameType.ANY to MATCH
[ https://issues.apache.org/jira/browse/KAFKA-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510335#comment-16510335 ] ASF GitHub Bot commented on KAFKA-7010: --- big-andy-coates opened a new pull request #5205: KAFKA-7010: Rename ResourceNameType to PatternType URL: https://github.com/apache/kafka/pull/5205 Fix for [KAFKA-7010](https://issues.apache.org/jira/browse/KAFKA-7010). The initial PR for KIP-290 #5117 added a new `ResourceNameType`, which was initially a field on `Resource` and `ResourceFilter`. However, follow on PRs have now moved the name type fields to new `ResourcePattern` and `ResourcePatternFilter` classes. This means the old name is no longer valid and may be confusing. The PR looks to rename the class to a more intuitive `resource.PatternType`. @cmccabe also requested that the current `ANY` value for this class be renamed to avoid confusion. `PatternType.ANY` currently causes `ResourcePatternFilter` to bring back all ACLs that would affect the supplied resource, i.e. it brings back literal, wildcard ACLs, and also does pattern matching to work out which prefix acls would affect the resource. This is very different from the behaviour of `ResourceType.ANY`, which just means the filter ignores the type of resources. `ANY` is to be renamed to `MATCH` to disambiguate it from other `ANY` filter types. A new `ANY` will be added that works in the same way as others, i.e. it will cause the filter to ignore the pattern type, (but won't do any pattern matching). ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Rename ResourceNameType.ANY to MATCH > > > Key: KAFKA-7010 > URL: https://issues.apache.org/jira/browse/KAFKA-7010 > Project: Kafka > Issue Type: Sub-task > Components: core, security >Reporter: Andy Coates >Assignee: Andy Coates >Priority: Major > Fix For: 2.0.0 > > > Following on from the PR > [#5117|[https://github.com/apache/kafka/pull/5117]...] and discussions with > Colin McCabe... > The current ResourceNameType.ANY may be misleading as it performs pattern > matching for wildcard and prefixed bindings. Where as ResourceName.ANY just > brings back any resource name. > Renaming to ResourceNameType.MATCH and adding more Java doc should clear this > up. > Finally, ResourceNameType is no longer appropriate as the type is used in > ResourcePattern and ResourcePatternFilter. Hence rename to PatternType. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7007) Use JSON for /kafka-acl-extended-changes path
[ https://issues.apache.org/jira/browse/KAFKA-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-7007: --- Description: Relating to one of the outstanding work items in PR [#5117|[https://github.com/apache/kafka/pull/5117]...] Keep Literal ACLs on the old paths, using the old formats, to maintain backwards compatibility. Have Prefixed, and any latter types, go on new paths, using JSON, (old brokers are not aware of them). Add checks to reject any adminClient requests to add prefixed acls before the cluster is fully upgraded. was: Relating to one of the outstanding work items in PR [#5117|[https://github.com/apache/kafka/pull/5117]...] The above PR seeing ACL change notifications come through two paths. Change the code to use a single path, with a Json value that defines the resource-name-type of the changed binding. > Use JSON for /kafka-acl-extended-changes path > - > > Key: KAFKA-7007 > URL: https://issues.apache.org/jira/browse/KAFKA-7007 > Project: Kafka > Issue Type: Sub-task > Components: core, security >Reporter: Andy Coates >Assignee: Andy Coates >Priority: Major > Fix For: 2.0.0 > > > Relating to one of the outstanding work items in PR > [#5117|[https://github.com/apache/kafka/pull/5117]...] > > Keep Literal ACLs on the old paths, using the old formats, to maintain > backwards compatibility. > Have Prefixed, and any latter types, go on new paths, using JSON, (old > brokers are not aware of them). > Add checks to reject any adminClient requests to add prefixed acls before the > cluster is fully upgraded. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7007) Use JSON for /kafka-acl-extended-changes path
[ https://issues.apache.org/jira/browse/KAFKA-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-7007. Resolution: Fixed merged the PR to trunk and 2.0 branch. > Use JSON for /kafka-acl-extended-changes path > - > > Key: KAFKA-7007 > URL: https://issues.apache.org/jira/browse/KAFKA-7007 > Project: Kafka > Issue Type: Sub-task > Components: core, security >Reporter: Andy Coates >Assignee: Andy Coates >Priority: Major > Fix For: 2.0.0 > > > Relating to one of the outstanding work items in PR > [#5117|[https://github.com/apache/kafka/pull/5117]...] > > Keep Literal ACLs on the old paths, using the old formats, to maintain > backwards compatibility. > Have Prefixed, and any latter types, go on new paths, using JSON, (old > brokers are not aware of them). > Add checks to reject any adminClient requests to add prefixed acls before the > cluster is fully upgraded. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7007) Use JSON for /kafka-acl-extended-changes path
[ https://issues.apache.org/jira/browse/KAFKA-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-7007: --- Summary: Use JSON for /kafka-acl-extended-changes path (was: All ACL changes should use single /kafka-acl-changes path ) > Use JSON for /kafka-acl-extended-changes path > - > > Key: KAFKA-7007 > URL: https://issues.apache.org/jira/browse/KAFKA-7007 > Project: Kafka > Issue Type: Sub-task > Components: core, security >Reporter: Andy Coates >Assignee: Andy Coates >Priority: Major > Fix For: 2.0.0 > > > Relating to one of the outstanding work items in PR > [#5117|[https://github.com/apache/kafka/pull/5117]...] > > The above PR seeing ACL change notifications come through two paths. Change > the code to use a single path, with a Json value that defines the > resource-name-type of the changed binding. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7007) All ACL changes should use single /kafka-acl-changes path
[
https://issues.apache.org/jira/browse/KAFKA-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510297#comment-16510297
]
ASF GitHub Bot commented on KAFKA-7007:
---
junrao closed pull request #5161: KAFKA-7007: All ACL changes should use
single /kafka-acl-changes path
URL: https://github.com/apache/kafka/pull/5161
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/clients/src/main/java/org/apache/kafka/common/resource/ResourceNameType.java
b/clients/src/main/java/org/apache/kafka/common/resource/ResourceNameType.java
index 7aa72170de9..0e4fc0f2712 100644
---
a/clients/src/main/java/org/apache/kafka/common/resource/ResourceNameType.java
+++
b/clients/src/main/java/org/apache/kafka/common/resource/ResourceNameType.java
@@ -62,6 +62,12 @@
.collect(Collectors.toMap(ResourceNameType::code,
Function.identity()))
);
+private final static Map NAME_TO_VALUE =
+Collections.unmodifiableMap(
+Arrays.stream(ResourceNameType.values())
+.collect(Collectors.toMap(ResourceNameType::name,
Function.identity()))
+);
+
private final byte code;
ResourceNameType(byte code) {
@@ -88,4 +94,11 @@ public boolean isUnknown() {
public static ResourceNameType fromCode(byte code) {
return CODE_TO_VALUE.getOrDefault(code, UNKNOWN);
}
+
+/**
+ * Return the ResourceNameType with the provided name or {@link #UNKNOWN}
if one cannot be found.
+ */
+public static ResourceNameType fromString(String name) {
+return NAME_TO_VALUE.getOrDefault(name, UNKNOWN);
+}
}
diff --git
a/core/src/main/scala/kafka/common/ZkNodeChangeNotificationListener.scala
b/core/src/main/scala/kafka/common/ZkNodeChangeNotificationListener.scala
index 51798519cb1..8ec7f953431 100644
--- a/core/src/main/scala/kafka/common/ZkNodeChangeNotificationListener.scala
+++ b/core/src/main/scala/kafka/common/ZkNodeChangeNotificationListener.scala
@@ -24,6 +24,8 @@ import kafka.zk.{KafkaZkClient, StateChangeHandlers}
import kafka.zookeeper.{StateChangeHandler, ZNodeChildChangeHandler}
import org.apache.kafka.common.utils.Time
+import scala.util.{Failure, Try}
+
/**
* Handle the notificationMessage.
*/
@@ -83,12 +85,7 @@ class ZkNodeChangeNotificationListener(private val zkClient:
KafkaZkClient,
for (notification <- notifications) {
val changeId = changeNumber(notification)
if (changeId > lastExecutedChange) {
-val changeZnode = seqNodeRoot + "/" + notification
-val (data, _) = zkClient.getDataAndStat(changeZnode)
-data match {
- case Some(d) => notificationHandler.processNotification(d)
- case None => warn(s"read null data from $changeZnode when
processing notification $notification")
-}
+processNotification(notification)
lastExecutedChange = changeId
}
}
@@ -100,6 +97,18 @@ class ZkNodeChangeNotificationListener(private val
zkClient: KafkaZkClient,
}
}
+ private def processNotification(notification: String): Unit = {
+val changeZnode = seqNodeRoot + "/" + notification
+val (data, _) = zkClient.getDataAndStat(changeZnode)
+data match {
+ case Some(d) => Try(notificationHandler.processNotification(d)) match {
+case Failure(e) => error(s"error processing change notification from
$changeZnode", e)
+case _ =>
+ }
+ case None => warn(s"read null data from $changeZnode")
+}
+ }
+
private def addChangeNotification(): Unit = {
if (!isClosed.get && queue.peek() == null)
queue.put(new ChangeNotification)
diff --git a/core/src/main/scala/kafka/security/auth/Resource.scala
b/core/src/main/scala/kafka/security/auth/Resource.scala
index f07a11c7300..a3dd06528f1 100644
--- a/core/src/main/scala/kafka/security/auth/Resource.scala
+++ b/core/src/main/scala/kafka/security/auth/Resource.scala
@@ -16,6 +16,7 @@
*/
package kafka.security.auth
+import kafka.common.KafkaException
import org.apache.kafka.common.resource.{ResourceNameType, ResourcePattern}
object Resource {
@@ -26,16 +27,18 @@ object Resource {
val WildCardResource = "*"
def fromString(str: String): Resource = {
-ResourceNameType.values.find(nameType => str.startsWith(nameType.name))
match {
- case Some(nameType) =>
-str.split(Separator, 3) match {
- case Array(_, resourceType, name, _*) => new
Resource(ResourceType.fromString(resourceType), name, nameType)
- case _ => throw new IllegalArgumentException("expected a string in
format ResourceType:ResourceName but got " + str)
-}
- case _ =>
[jira] [Commented] (KAFKA-6977) Kafka Streams - java.lang.IllegalStateException: Unexpected error code 2 while fetching data
[
https://issues.apache.org/jira/browse/KAFKA-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510295#comment-16510295
]
Eugen Feller commented on KAFKA-6977:
-
Quick update. I have tried with 1.1.0 client and same issue.
> Kafka Streams - java.lang.IllegalStateException: Unexpected error code 2
> while fetching data
> -
>
> Key: KAFKA-6977
> URL: https://issues.apache.org/jira/browse/KAFKA-6977
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.1
>Reporter: Eugen Feller
>Priority: Blocker
> Labels: streams
>
> We are running Kafka Streams 0.11.0.1 with Kafka Broker 0.10.2.1 and
> constantly run into the following exception:
> {code:java}
> [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1]
> INFO org.apache.kafka.streams.processor.internals.StreamThread -
> stream-thread
> [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1]
> partition assignment took 40 ms.
> current active tasks: [0_0, 0_1, 0_3, 0_4, 0_5, 0_10, 0_12, 0_13, 0_14, 0_15,
> 0_17, 0_20, 0_21, 0_23, 0_25, 0_28, 0_2, 0_6, 0_7, 0_8, 0_9, 0_11, 0_16,
> 0_18, 0_19, 0_22, 0_24, 0_26, 0_27, 0_29, 0_30, 0_31]
> current standby tasks: []
> previous active tasks: [0_0, 0_1, 0_3, 0_4, 0_5, 0_10, 0_12, 0_13, 0_14,
> 0_15, 0_17, 0_20, 0_21, 0_23, 0_25, 0_28]
> [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1]
> INFO org.apache.kafka.streams.processor.internals.StreamThread -
> stream-thread
> [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1]
> State transition from PARTITIONS_ASSIGNED to RUNNING.
> [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1]
> INFO org.apache.kafka.streams.KafkaStreams - stream-client
> [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e] State
> transition from REBALANCING to RUNNING.
> [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1]
> ERROR org.apache.kafka.streams.processor.internals.StreamThread -
> stream-thread
> [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1]
> Encountered the following error during processing:
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:886)
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:525)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1086)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:536)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:490)
> java.lang.IllegalStateException: Unexpected error code 2 while fetching data
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> INFO org.apache.kafka.streams.processor.internals.StreamThread -
> stream-thread
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> Shutting down
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> INFO org.apache.kafka.streams.processor.internals.StreamThread -
> stream-thread
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> State transition from RUNNING to PENDING_SHUTDOWN.
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> INFO org.apache.kafka.clients.producer.KafkaProducer - Closing the Kafka
> producer with timeoutMillis = 9223372036854775807 ms.
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> INFO org.apache.kafka.streams.processor.internals.StreamThread -
> stream-thread
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> Stream thread shutdown complete
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> INFO org.apache.kafka.streams.processor.internals.StreamThread -
> stream-thread
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> State transition from PENDING_SHUTDOWN to DEAD.
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> INFO org.apache.kafka.streams.KafkaStreams - stream-client
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4] State
> transition from RUNNING to ERROR.
> [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1]
> WARN
[jira] [Commented] (KAFKA-7047) Connect isolation whitelist does not include SimpleHeaderConverter
[ https://issues.apache.org/jira/browse/KAFKA-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510198#comment-16510198 ] ASF GitHub Bot commented on KAFKA-7047: --- rhauch opened a new pull request #5204: KAFKA-7047: Added SimpleHeaderConverter to plugin isolation whitelist URL: https://github.com/apache/kafka/pull/5204 This was originally missed when headers were added as part of KIP-145 in AK 1.1. An additional unit test was added in line with the StringConverter. This should be backported to the AK `1.1` branch so that it is included in the next bugfix release. The `SimpleHeaderConverter` class that we're referencing was first added in the 1.1.0 release, so there's no reason to backport earlier. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Connect isolation whitelist does not include SimpleHeaderConverter > -- > > Key: KAFKA-7047 > URL: https://issues.apache.org/jira/browse/KAFKA-7047 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 1.1.0 >Reporter: Randall Hauch >Assignee: Randall Hauch >Priority: Critical > > The SimpleHeaderConverter added in 1.1.0 was never added to the PluginUtils > whitelist so that this header converter is loaded in isolation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6782) GlobalKTable GlobalStateStore never finishes restoring when consuming aborted messages
[
https://issues.apache.org/jira/browse/KAFKA-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax resolved KAFKA-6782.
Resolution: Fixed
Fix Version/s: 1.1.1
1.0.2
0.11.0.3
2.0.0
> GlobalKTable GlobalStateStore never finishes restoring when consuming aborted
> messages
> --
>
> Key: KAFKA-6782
> URL: https://issues.apache.org/jira/browse/KAFKA-6782
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.1.0, 1.0.1
>Reporter: Lingxiao WANG
>Assignee: Lingxiao WANG
>Priority: Major
> Fix For: 2.0.0, 0.11.0.3, 1.0.2, 1.1.1
>
>
> Same problem with https://issues.apache.org/jira/browse/KAFKA-6190, but his
> solution which is below, works for the succeed transactional messages. But
> when there are aborted messages, it will be in infinite loop. Here is his
> proposition :
> {code:java}
> while (offset < highWatermark) {
> final ConsumerRecords records = consumer.poll(100);
> for (ConsumerRecord record : records) {
> if (record.key() != null) {
>stateRestoreCallback.restore(record.key(), record.value());
> }
> offset = consumer.position(topicPartition);
> }
> }{code}
> Concretely, when the consumer consume a set of aborted messages, it polls 0
> records, and the code 'offset = consumer.position(topicPartition)' doesn't
> have any opportunity to execute.
> So I propose to move the code 'offset = consumer.position(topicPartition)'
> outside of the cycle to guarantee that event if no records are polled, the
> offset can always be updated.
> {code:java}
> while (offset < highWatermark) {
> final ConsumerRecords records = consumer.poll(100);
> for (ConsumerRecord record : records) {
> if (record.key() != null) {
>stateRestoreCallback.restore(record.key(), record.value());
> }
> }
> offset = consumer.position(topicPartition);
> }{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7044) kafka-consumer-groups.sh NullPointerException describing round robin or sticky assignors
[
https://issues.apache.org/jira/browse/KAFKA-7044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510156#comment-16510156
]
Vahid Hashemian commented on KAFKA-7044:
[~jfield] I have not been able to reproduce the error you reported after
running a couple of consumers with either of these assignors. Can you provide
the steps to reproduce?
> kafka-consumer-groups.sh NullPointerException describing round robin or
> sticky assignors
>
>
> Key: KAFKA-7044
> URL: https://issues.apache.org/jira/browse/KAFKA-7044
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Affects Versions: 1.1.0
> Environment: CentOS 7.4, Oracle JDK 1.8
>Reporter: Jeff Field
>Assignee: Vahid Hashemian
>Priority: Minor
>
> We've recently moved to using the round robin assignor for one of our
> consumer groups, and started testing the sticky assignor. In both cases,
> using Kafka 1.1.0 we get a null pointer exception *unless* the group being
> described is rebalancing:
> {code:java}
> kafka-consumer-groups --bootstrap-server fqdn:9092 --describe --group
> groupname-for-consumer
> Error: Executing consumer group command failed due to null
> [2018-06-12 01:32:34,179] DEBUG Exception in consumer group command
> (kafka.admin.ConsumerGroupCommand$)
> java.lang.NullPointerException
> at scala.Predef$.Long2long(Predef.scala:363)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$getLogEndOffsets$2.apply(ConsumerGroupCommand.scala:612)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$getLogEndOffsets$2.apply(ConsumerGroupCommand.scala:610)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.immutable.List.map(List.scala:296)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.getLogEndOffsets(ConsumerGroupCommand.scala:610)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describePartitions(ConsumerGroupCommand.scala:328)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.collectConsumerAssignment(ConsumerGroupCommand.scala:308)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.collectConsumerAssignment(ConsumerGroupCommand.scala:544)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10$$anonfun$13.apply(ConsumerGroupCommand.scala:571)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10$$anonfun$13.apply(ConsumerGroupCommand.scala:565)
> at scala.collection.immutable.List.flatMap(List.scala:338)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10.apply(ConsumerGroupCommand.scala:565)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService$$anonfun$10.apply(ConsumerGroupCommand.scala:558)
> at scala.Option.map(Option.scala:146)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.collectGroupOffsets(ConsumerGroupCommand.scala:558)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describeGroup(ConsumerGroupCommand.scala:271)
> at
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:544)
> at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:77)
> at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
> [2018-06-12 01:32:34,255] DEBUG Removed sensor with name connections-closed:
> (org.apache.kafka.common.metrics.Metrics){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7012) Performance issue upgrading to kafka 1.0.1 or 1.1
[ https://issues.apache.org/jira/browse/KAFKA-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510122#comment-16510122 ] radai rosenblatt commented on KAFKA-7012: - in light of that i think the safe change to make would be to only record keys with bytes still buffered if memory pressure prevented anything from being read out this round (as opposed to always, as is current behaviour). > Performance issue upgrading to kafka 1.0.1 or 1.1 > - > > Key: KAFKA-7012 > URL: https://issues.apache.org/jira/browse/KAFKA-7012 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.1.0, 1.0.1 >Reporter: rajadayalan perumalsamy >Assignee: praveen >Priority: Major > Attachments: Commit-47ee8e954-0607-bufferkeys-nopoll-profile.png, > Commit-47ee8e954-0607-memory.png, Commit-47ee8e954-0607-profile.png, > Commit-47ee8e954-profile.png, Commit-47ee8e954-profile2.png, > Commit-f15cdbc91b-profile.png, Commit-f15cdbc91b-profile2.png > > > We are trying to upgrade kafka cluster from Kafka 0.11.0.1 to Kafka 1.0.1. > After upgrading 1 node on the cluster, we notice that network threads use > most of the cpu. It is a 3 node cluster with 15k messages/sec on each node. > With Kafka 0.11.0.1 typical usage of the servers is around 50 to 60% > vcpu(using less than 1 vcpu). After upgrade we are noticing that cpu usage is > high depending on the number of network threads used. If networks threads is > set to 8, then the cpu usage is around 850%(9 vcpus) and if it is set to 4 > then the cpu usage is around 450%(5 vcpus). Using the same kafka > server.properties for both. > Did further analysis with git bisect, couple of build and deploys, traced the > issue to commit 47ee8e954df62b9a79099e944ec4be29afe046f6. CPU usage is fine > for commit f15cdbc91b240e656d9a2aeb6877e94624b21f8d. But with commit > 47ee8e954df62b9a79099e944ec4be29afe046f6 cpu usage has increased. Have > attached screenshots of profiling done with both the commits. Screenshot > Commit-f15cdbc91b-profile shows less cpu usage by network threads and > Screenshots Commit-47ee8e954-profile and Commit-47ee8e954-profile2 show > higher cpu usage(almost entire cpu usage) by network threads. Also noticed > that kafka.network.Processor.poll() method is invoked 10 times more with > commit 47ee8e954df62b9a79099e944ec4be29afe046f6. > We need the issue to be resolved to upgrade the cluster. Please let me know > if you need any additional information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-5630) Consumer poll loop over the same record after a CorruptRecordException
[ https://issues.apache.org/jira/browse/KAFKA-5630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510050#comment-16510050 ] Eugen Feller edited comment on KAFKA-5630 at 6/12/18 7:10 PM: -- [~becket_qin] [~hachikuji] Looks like we are running into a similar issue using 0.10.2.1 broker and kafka streams client 0.11.0.1. Wonder if this fix helps only if broker is also on 0.11.01? This is the my related JIRA (https://issues.apache.org/jira/browse/KAFKA-6977) Thanks. was (Author: efeller): [~becket_qin] Looks like we are running into a similar issue using 0.10.2.1 broker and kafka streams client 0.11.0.1. Wonder if this fix helps only if broker is also on 0.11.01? This is the my related JIRA (https://issues.apache.org/jira/browse/KAFKA-6977) Thanks. > Consumer poll loop over the same record after a CorruptRecordException > -- > > Key: KAFKA-5630 > URL: https://issues.apache.org/jira/browse/KAFKA-5630 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.11.0.0 >Reporter: Vincent Maurin >Assignee: Jiangjie Qin >Priority: Critical > Labels: regression, reliability > Fix For: 0.11.0.1, 1.0.0 > > > Hello > While consuming a topic with log compaction enabled, I am getting an infinite > consumption loop of the same record, i.e, each call to poll is returning to > me 500 times one record (500 is my max.poll.records). I am using the java > client 0.11.0.0. > Running the code with the debugger, the initial problem come from > `Fetcher.PartitionRecords,fetchRecords()`. > Here I get a `org.apache.kafka.common.errors.CorruptRecordException: Record > size is less than the minimum record overhead (14)` > Then the boolean `hasExceptionInLastFetch` is set to true, resulting the test > block in `Fetcher.PartitionRecords.nextFetchedRecord()` to always return the > last record. > I guess the corruption problem is similar too > https://issues.apache.org/jira/browse/KAFKA-5582 but this behavior of the > client is probably not the expected one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5630) Consumer poll loop over the same record after a CorruptRecordException
[ https://issues.apache.org/jira/browse/KAFKA-5630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510050#comment-16510050 ] Eugen Feller commented on KAFKA-5630: - Looks like we are running into a similar issue using 0.10.2.1 broker and kafka streams client 0.11.0.1. Wonder if this fix helps only if broker is also on 0.11.01? This is the my related JIRA (https://issues.apache.org/jira/browse/KAFKA-6977) Thanks. > Consumer poll loop over the same record after a CorruptRecordException > -- > > Key: KAFKA-5630 > URL: https://issues.apache.org/jira/browse/KAFKA-5630 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.11.0.0 >Reporter: Vincent Maurin >Assignee: Jiangjie Qin >Priority: Critical > Labels: regression, reliability > Fix For: 0.11.0.1, 1.0.0 > > > Hello > While consuming a topic with log compaction enabled, I am getting an infinite > consumption loop of the same record, i.e, each call to poll is returning to > me 500 times one record (500 is my max.poll.records). I am using the java > client 0.11.0.0. > Running the code with the debugger, the initial problem come from > `Fetcher.PartitionRecords,fetchRecords()`. > Here I get a `org.apache.kafka.common.errors.CorruptRecordException: Record > size is less than the minimum record overhead (14)` > Then the boolean `hasExceptionInLastFetch` is set to true, resulting the test > block in `Fetcher.PartitionRecords.nextFetchedRecord()` to always return the > last record. > I guess the corruption problem is similar too > https://issues.apache.org/jira/browse/KAFKA-5582 but this behavior of the > client is probably not the expected one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-5630) Consumer poll loop over the same record after a CorruptRecordException
[ https://issues.apache.org/jira/browse/KAFKA-5630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510050#comment-16510050 ] Eugen Feller edited comment on KAFKA-5630 at 6/12/18 7:10 PM: -- [~becket_qin] Looks like we are running into a similar issue using 0.10.2.1 broker and kafka streams client 0.11.0.1. Wonder if this fix helps only if broker is also on 0.11.01? This is the my related JIRA (https://issues.apache.org/jira/browse/KAFKA-6977) Thanks. was (Author: efeller): Looks like we are running into a similar issue using 0.10.2.1 broker and kafka streams client 0.11.0.1. Wonder if this fix helps only if broker is also on 0.11.01? This is the my related JIRA (https://issues.apache.org/jira/browse/KAFKA-6977) Thanks. > Consumer poll loop over the same record after a CorruptRecordException > -- > > Key: KAFKA-5630 > URL: https://issues.apache.org/jira/browse/KAFKA-5630 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.11.0.0 >Reporter: Vincent Maurin >Assignee: Jiangjie Qin >Priority: Critical > Labels: regression, reliability > Fix For: 0.11.0.1, 1.0.0 > > > Hello > While consuming a topic with log compaction enabled, I am getting an infinite > consumption loop of the same record, i.e, each call to poll is returning to > me 500 times one record (500 is my max.poll.records). I am using the java > client 0.11.0.0. > Running the code with the debugger, the initial problem come from > `Fetcher.PartitionRecords,fetchRecords()`. > Here I get a `org.apache.kafka.common.errors.CorruptRecordException: Record > size is less than the minimum record overhead (14)` > Then the boolean `hasExceptionInLastFetch` is set to true, resulting the test > block in `Fetcher.PartitionRecords.nextFetchedRecord()` to always return the > last record. > I guess the corruption problem is similar too > https://issues.apache.org/jira/browse/KAFKA-5582 but this behavior of the > client is probably not the expected one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-6782) GlobalKTable GlobalStateStore never finishes restoring when consuming aborted messages
[
https://issues.apache.org/jira/browse/KAFKA-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reassigned KAFKA-6782:
--
Assignee: Lingxiao WANG
> GlobalKTable GlobalStateStore never finishes restoring when consuming aborted
> messages
> --
>
> Key: KAFKA-6782
> URL: https://issues.apache.org/jira/browse/KAFKA-6782
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.1.0, 1.0.1
>Reporter: Lingxiao WANG
>Assignee: Lingxiao WANG
>Priority: Major
>
> Same problem with https://issues.apache.org/jira/browse/KAFKA-6190, but his
> solution which is below, works for the succeed transactional messages. But
> when there are aborted messages, it will be in infinite loop. Here is his
> proposition :
> {code:java}
> while (offset < highWatermark) {
> final ConsumerRecords records = consumer.poll(100);
> for (ConsumerRecord record : records) {
> if (record.key() != null) {
>stateRestoreCallback.restore(record.key(), record.value());
> }
> offset = consumer.position(topicPartition);
> }
> }{code}
> Concretely, when the consumer consume a set of aborted messages, it polls 0
> records, and the code 'offset = consumer.position(topicPartition)' doesn't
> have any opportunity to execute.
> So I propose to move the code 'offset = consumer.position(topicPartition)'
> outside of the cycle to guarantee that event if no records are polled, the
> offset can always be updated.
> {code:java}
> while (offset < highWatermark) {
> final ConsumerRecords records = consumer.poll(100);
> for (ConsumerRecord record : records) {
> if (record.key() != null) {
>stateRestoreCallback.restore(record.key(), record.value());
> }
> }
> offset = consumer.position(topicPartition);
> }{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6782) GlobalKTable GlobalStateStore never finishes restoring when consuming aborted messages
[
https://issues.apache.org/jira/browse/KAFKA-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510029#comment-16510029
]
ASF GitHub Bot commented on KAFKA-6782:
---
mjsax closed pull request #4900: KAFKA-6782: solved the bug of restoration of
aborted messages for GlobalStateStore and KGlobalTable
URL: https://github.com/apache/kafka/pull/4900
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.gitignore b/.gitignore
index 04f8feed0ad..fe191eed44b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,6 @@
dist
*classes
+*.class
target/
build/
build_eclipse/
diff --git a/kafka b/kafka
new file mode 16
index 000..cc43e77bbbf
--- /dev/null
+++ b/kafka
@@ -0,0 +1 @@
+Subproject commit cc43e77bbbfad71883011186de55603c936cbcd1
diff --git
a/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImpl.java
b/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImpl.java
index e8ec5e9fe5f..96064b6d4ad 100644
---
a/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImpl.java
+++
b/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImpl.java
@@ -268,8 +268,8 @@ private void restoreState(final StateRestoreCallback
stateRestoreCallback,
if (record.key() != null) {
restoreRecords.add(KeyValue.pair(record.key(),
record.value()));
}
-offset = globalConsumer.position(topicPartition);
}
+offset = globalConsumer.position(topicPartition);
stateRestoreAdapter.restoreAll(restoreRecords);
stateRestoreListener.onBatchRestored(topicPartition,
storeName, offset, restoreRecords.size());
restoreCount += restoreRecords.size();
diff --git
a/streams/src/test/java/org/apache/kafka/streams/integration/GlobalKTableEOSIntegrationTest.java
b/streams/src/test/java/org/apache/kafka/streams/integration/GlobalKTableEOSIntegrationTest.java
new file mode 100644
index 000..f7c0e55c05e
--- /dev/null
+++
b/streams/src/test/java/org/apache/kafka/streams/integration/GlobalKTableEOSIntegrationTest.java
@@ -0,0 +1,390 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import kafka.utils.MockTime;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.errors.InvalidStateStoreException;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.ForeachAction;
+import org.apache.kafka.streams.kstream.GlobalKTable;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KeyValueMapper;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.ValueJoiner;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.QueryableStoreTypes;
+import org.apache.kafka.streams.state.ReadOnlyKeyValueStore;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestCondition;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
[jira] [Commented] (KAFKA-7050) Decrease consumer request timeout to 30s
[ https://issues.apache.org/jira/browse/KAFKA-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510013#comment-16510013 ] ASF GitHub Bot commented on KAFKA-7050: --- hachikuji opened a new pull request #5203: KAFKA-7050: Decrease default consumer request timeout to 30s URL: https://github.com/apache/kafka/pull/5203 This patch changes the default `request.timeout.ms` of the consumer to 30 seconds. Additionally, it adds logic to `NetworkClient` to set timeouts at the request level. We use this to handle the special case of the JoinGroup request, which may block for as long as the value configured by `max.poll.interval.ms`. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Decrease consumer request timeout to 30s > > > Key: KAFKA-7050 > URL: https://issues.apache.org/jira/browse/KAFKA-7050 > Project: Kafka > Issue Type: Improvement >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Major > > Per KIP-266 discussion, we should lower the request timeout. We should also > add new logic to override this timeout for the JoinGroup request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7050) Decrease consumer request timeout to 30s
Jason Gustafson created KAFKA-7050: -- Summary: Decrease consumer request timeout to 30s Key: KAFKA-7050 URL: https://issues.apache.org/jira/browse/KAFKA-7050 Project: Kafka Issue Type: Improvement Reporter: Jason Gustafson Assignee: Jason Gustafson Per KIP-266 discussion, we should lower the request timeout. We should also add new logic to override this timeout for the JoinGroup request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7049) InternalTopicIntegrationTest sometimes fails
Ted Yu created KAFKA-7049:
-
Summary: InternalTopicIntegrationTest sometimes fails
Key: KAFKA-7049
URL: https://issues.apache.org/jira/browse/KAFKA-7049
Project: Kafka
Issue Type: Test
Reporter: Ted Yu
Saw the following based on commit fa1d0383902260576132e09bdf9efcc2784b55b4 :
{code}
org.apache.kafka.streams.integration.InternalTopicIntegrationTest >
shouldCompactTopicsForKeyValueStoreChangelogs FAILED
java.lang.RuntimeException: Timed out waiting for completion.
lagMetrics=[0/2] totalLag=[0.0]
at
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForCompletion(IntegrationTestUtils.java:227)
at
org.apache.kafka.streams.integration.InternalTopicIntegrationTest.shouldCompactTopicsForKeyValueStoreChangelogs(InternalTopicIntegrationTest.java:164)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7023) Kafka Streams RocksDB bulk loading config may not be honored with customized RocksDBConfigSetter
[
https://issues.apache.org/jira/browse/KAFKA-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509982#comment-16509982
]
ASF GitHub Bot commented on KAFKA-7023:
---
guozhangwang closed pull request #5197: KAFKA-7023: Add unit test
URL: https://github.com/apache/kafka/pull/5197
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java
b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java
index 6084ecbf1e0..e858ac07921 100644
---
a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java
+++
b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java
@@ -572,4 +572,9 @@ public void onRestoreEnd(final TopicPartition
topicPartition,
rocksDBStore.toggleDbForBulkLoading(false);
}
}
+
+// for testing
+public Options getOptions() {
+return options;
+}
}
diff --git
a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBStoreTest.java
b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBStoreTest.java
index b7a9d375d9c..63d877af5d6 100644
---
a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBStoreTest.java
+++
b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBStoreTest.java
@@ -16,23 +16,21 @@
*/
package org.apache.kafka.streams.state.internals;
-import org.apache.kafka.common.metrics.Metrics;
import org.apache.kafka.common.serialization.Deserializer;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.serialization.Serializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.common.utils.Bytes;
-import org.apache.kafka.common.utils.LogContext;
import org.apache.kafka.common.utils.Utils;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.errors.ProcessorStateException;
-import org.apache.kafka.streams.processor.internals.MockStreamsMetrics;
+import org.apache.kafka.streams.processor.StateRestoreListener;
import org.apache.kafka.streams.state.KeyValueIterator;
import org.apache.kafka.streams.state.RocksDBConfigSetter;
import org.apache.kafka.test.InternalMockProcessorContext;
-import org.apache.kafka.test.NoOpRecordCollector;
+import org.apache.kafka.test.StreamsTestUtils;
import org.apache.kafka.test.TestUtils;
import org.junit.After;
import org.junit.Before;
@@ -43,10 +41,10 @@
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
-import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Properties;
import java.util.Set;
import static org.hamcrest.CoreMatchers.equalTo;
@@ -57,8 +55,6 @@
import static org.junit.Assert.fail;
public class RocksDBStoreTest {
-private final File tempDir = TestUtils.tempDirectory();
-
private Serializer stringSerializer = new StringSerializer();
private Deserializer stringDeserializer = new StringDeserializer();
private RocksDBStore rocksDBStore;
@@ -67,13 +63,14 @@
@Before
public void setUp() {
+final Properties props = StreamsTestUtils.minimalStreamsConfig();
+props.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG,
MockRocksDbConfigSetter.class);
rocksDBStore = new RocksDBStore("test");
dir = TestUtils.tempDirectory();
context = new InternalMockProcessorContext(dir,
Serdes.String(),
Serdes.String(),
-new NoOpRecordCollector(),
-new ThreadCache(new LogContext("testCache "), 0, new
MockStreamsMetrics(new Metrics(;
+new StreamsConfig(props));
}
@After
@@ -81,6 +78,21 @@ public void tearDown() {
rocksDBStore.close();
}
+@Test
+public void shouldRespectBulkloadOptionsDuringInit() {
+rocksDBStore.init(context, rocksDBStore);
+
+StateRestoreListener restoreListener =
context.getRestoreListener(rocksDBStore.name());
+
+restoreListener.onRestoreStart(null, rocksDBStore.name(), 0L, 0L);
+
+assertThat(rocksDBStore.getOptions().level0FileNumCompactionTrigger(),
equalTo(1 << 30));
+
+restoreListener.onRestoreEnd(null, rocksDBStore.name(), 0L);
+
+assertThat(rocksDBStore.getOptions().level0FileNumCompactionTrigger(),
equalTo(10));
+}
+
@Test
public void
shouldNotThrowExceptionOnRestoreWhenThereIsPreExistingRocksDbFiles()
[jira] [Resolved] (KAFKA-6976) Kafka Streams instances going in to DEAD state
[
https://issues.apache.org/jira/browse/KAFKA-6976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax resolved KAFKA-6976.
Resolution: Not A Bug
Closing this because it's not a bug, but a configuration issue.
> Kafka Streams instances going in to DEAD state
> --
>
> Key: KAFKA-6976
> URL: https://issues.apache.org/jira/browse/KAFKA-6976
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Deepak Goyal
>Priority: Blocker
> Attachments: kafkaStreamsDeadState.log
>
>
> We are using Kafka 0.10.2.0, Kafka-Streams 1.1.0. We have Kafka Cluster of 16
> machines, and topic that is being consumed by Kafka Streams has 256
> partitions. We spawned 400 machines of Kakfa Streams application. We see that
> all of the StreamThreads go in to DEAD state.
> {quote}{{[2018-05-25 05:59:29,282] INFO stream-thread
> [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1] State transition
> from PENDING_SHUTDOWN to DEAD
> (org.apache.kafka.streams.processor.internals.StreamThread) [2018-05-25
> 05:59:29,282] INFO stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7]
> State transition from REBALANCING to ERROR
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] WARN
> stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] INFO
> stream-thread [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1]
> Shutdown complete
> (org.apache.kafka.streams.processor.internals.StreamThread)}}
> {quote}
> Please note that when we only have 100 kafka-streams application machines,
> things are working as expected. We see that instances are consuming messages
> from topic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6976) Kafka Streams instances going in to DEAD state
[
https://issues.apache.org/jira/browse/KAFKA-6976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509977#comment-16509977
]
Matthias J. Sax commented on KAFKA-6976:
Glad it works not. Can you do a PR to update the docs?
> Kafka Streams instances going in to DEAD state
> --
>
> Key: KAFKA-6976
> URL: https://issues.apache.org/jira/browse/KAFKA-6976
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Deepak Goyal
>Priority: Blocker
> Attachments: kafkaStreamsDeadState.log
>
>
> We are using Kafka 0.10.2.0, Kafka-Streams 1.1.0. We have Kafka Cluster of 16
> machines, and topic that is being consumed by Kafka Streams has 256
> partitions. We spawned 400 machines of Kakfa Streams application. We see that
> all of the StreamThreads go in to DEAD state.
> {quote}{{[2018-05-25 05:59:29,282] INFO stream-thread
> [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1] State transition
> from PENDING_SHUTDOWN to DEAD
> (org.apache.kafka.streams.processor.internals.StreamThread) [2018-05-25
> 05:59:29,282] INFO stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7]
> State transition from REBALANCING to ERROR
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] WARN
> stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] INFO
> stream-thread [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1]
> Shutdown complete
> (org.apache.kafka.streams.processor.internals.StreamThread)}}
> {quote}
> Please note that when we only have 100 kafka-streams application machines,
> things are working as expected. We see that instances are consuming messages
> from topic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-7012) Performance issue upgrading to kafka 1.0.1 or 1.1
[ https://issues.apache.org/jira/browse/KAFKA-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509927#comment-16509927 ] rajadayalan perumalsamy edited comment on KAFKA-7012 at 6/12/18 6:07 PM: - Tried out in a different cluster 20 nodes(million msgs/sec), upgraded 5 nodes to 1.0.1 with the patched jar. can confirm that it is not affecting function/Missing msgs and latency also looks fine. was (Author: rajadayalanvdms): Tried out the patched jar in a different cluster 20 nodes(million msgs/sec), upgraded 5 nodes to 1.0.1 with the patched jar. can confirm that it is not affecting function/Missing msgs and latency also looks fine. > Performance issue upgrading to kafka 1.0.1 or 1.1 > - > > Key: KAFKA-7012 > URL: https://issues.apache.org/jira/browse/KAFKA-7012 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.1.0, 1.0.1 >Reporter: rajadayalan perumalsamy >Assignee: praveen >Priority: Major > Attachments: Commit-47ee8e954-0607-bufferkeys-nopoll-profile.png, > Commit-47ee8e954-0607-memory.png, Commit-47ee8e954-0607-profile.png, > Commit-47ee8e954-profile.png, Commit-47ee8e954-profile2.png, > Commit-f15cdbc91b-profile.png, Commit-f15cdbc91b-profile2.png > > > We are trying to upgrade kafka cluster from Kafka 0.11.0.1 to Kafka 1.0.1. > After upgrading 1 node on the cluster, we notice that network threads use > most of the cpu. It is a 3 node cluster with 15k messages/sec on each node. > With Kafka 0.11.0.1 typical usage of the servers is around 50 to 60% > vcpu(using less than 1 vcpu). After upgrade we are noticing that cpu usage is > high depending on the number of network threads used. If networks threads is > set to 8, then the cpu usage is around 850%(9 vcpus) and if it is set to 4 > then the cpu usage is around 450%(5 vcpus). Using the same kafka > server.properties for both. > Did further analysis with git bisect, couple of build and deploys, traced the > issue to commit 47ee8e954df62b9a79099e944ec4be29afe046f6. CPU usage is fine > for commit f15cdbc91b240e656d9a2aeb6877e94624b21f8d. But with commit > 47ee8e954df62b9a79099e944ec4be29afe046f6 cpu usage has increased. Have > attached screenshots of profiling done with both the commits. Screenshot > Commit-f15cdbc91b-profile shows less cpu usage by network threads and > Screenshots Commit-47ee8e954-profile and Commit-47ee8e954-profile2 show > higher cpu usage(almost entire cpu usage) by network threads. Also noticed > that kafka.network.Processor.poll() method is invoked 10 times more with > commit 47ee8e954df62b9a79099e944ec4be29afe046f6. > We need the issue to be resolved to upgrade the cluster. Please let me know > if you need any additional information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7012) Performance issue upgrading to kafka 1.0.1 or 1.1
[ https://issues.apache.org/jira/browse/KAFKA-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509927#comment-16509927 ] rajadayalan perumalsamy commented on KAFKA-7012: Tried out the patched jar in a different cluster 20 nodes(million msgs/sec), upgraded 5 nodes to 1.0.1 with the patched jar. can confirm that it is not affecting function/Missing msgs and latency also looks fine. > Performance issue upgrading to kafka 1.0.1 or 1.1 > - > > Key: KAFKA-7012 > URL: https://issues.apache.org/jira/browse/KAFKA-7012 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.1.0, 1.0.1 >Reporter: rajadayalan perumalsamy >Assignee: praveen >Priority: Major > Attachments: Commit-47ee8e954-0607-bufferkeys-nopoll-profile.png, > Commit-47ee8e954-0607-memory.png, Commit-47ee8e954-0607-profile.png, > Commit-47ee8e954-profile.png, Commit-47ee8e954-profile2.png, > Commit-f15cdbc91b-profile.png, Commit-f15cdbc91b-profile2.png > > > We are trying to upgrade kafka cluster from Kafka 0.11.0.1 to Kafka 1.0.1. > After upgrading 1 node on the cluster, we notice that network threads use > most of the cpu. It is a 3 node cluster with 15k messages/sec on each node. > With Kafka 0.11.0.1 typical usage of the servers is around 50 to 60% > vcpu(using less than 1 vcpu). After upgrade we are noticing that cpu usage is > high depending on the number of network threads used. If networks threads is > set to 8, then the cpu usage is around 850%(9 vcpus) and if it is set to 4 > then the cpu usage is around 450%(5 vcpus). Using the same kafka > server.properties for both. > Did further analysis with git bisect, couple of build and deploys, traced the > issue to commit 47ee8e954df62b9a79099e944ec4be29afe046f6. CPU usage is fine > for commit f15cdbc91b240e656d9a2aeb6877e94624b21f8d. But with commit > 47ee8e954df62b9a79099e944ec4be29afe046f6 cpu usage has increased. Have > attached screenshots of profiling done with both the commits. Screenshot > Commit-f15cdbc91b-profile shows less cpu usage by network threads and > Screenshots Commit-47ee8e954-profile and Commit-47ee8e954-profile2 show > higher cpu usage(almost entire cpu usage) by network threads. Also noticed > that kafka.network.Processor.poll() method is invoked 10 times more with > commit 47ee8e954df62b9a79099e944ec4be29afe046f6. > We need the issue to be resolved to upgrade the cluster. Please let me know > if you need any additional information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7048) NPE when creating connector
[
https://issues.apache.org/jira/browse/KAFKA-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509911#comment-16509911
]
ASF GitHub Bot commented on KAFKA-7048:
---
chia7712 opened a new pull request #5202: KAFKA-7048 NPE when creating connector
URL: https://github.com/apache/kafka/pull/5202
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> NPE when creating connector
> ---
>
> Key: KAFKA-7048
> URL: https://issues.apache.org/jira/browse/KAFKA-7048
> Project: Kafka
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 2.0.0
>
>
> KAFKA-6886 introduced the ConfigTransformer to transform the given
> configuration data. ConfigTransformer#transform(Map) expect
> the passed config won't be null but DistributedHerder#putConnectorConfig call
> the #transform before updating the snapshot (see below). Hence, it cause the
> NPE.
> {code:java}
> // Note that we use the updated connector config despite the fact that we
> don't have an updated
> // snapshot yet. The existing task info should still be accurate.
> Map map = configState.connectorConfig(connName);
> ConnectorInfo info = new ConnectorInfo(connName, config,
> configState.tasks(connName),
> map == null ? null :
> connectorTypeForClass(map.get(ConnectorConfig.CONNECTOR_CLASS_CONFIG)));
> callback.onCompletion(null, new Created<>(!exists, info));
> return null;{code}
> We can add a null check to "configs" (see below) to resolve the NPE. It means
> we WON'T pass the null configs to configTransformer
> {code:java}
> public Map connectorConfig(String connector) {
> Map configs = connectorConfigs.get(connector);
> if (configTransformer != null) { // add a condition "configs != null"
> configs = configTransformer.transform(connector, configs);
> }
> return configs;
> }{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6711) GlobalStateManagerImpl should not write offsets of in-memory stores in checkpoint file
[ https://issues.apache.org/jira/browse/KAFKA-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509909#comment-16509909 ] Matthias J. Sax commented on KAFKA-6711: I am talking this over to get the fix into 2.0 release. > GlobalStateManagerImpl should not write offsets of in-memory stores in > checkpoint file > -- > > Key: KAFKA-6711 > URL: https://issues.apache.org/jira/browse/KAFKA-6711 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 1.0.1 >Reporter: Cemalettin Koç >Assignee: Cemalettin Koç >Priority: Major > Labels: newbie > > We are using an InMemoryStore along with GlobalKTable and I noticed that > after each restart I am losing all my data. When I debug it, > `/tmp/kafka-streams/category-client-1/global/.checkpoint` file contains > offset for my GlobalKTable topic. I had checked GlobalStateManagerImpl > implementation and noticed that it is not guarded for cases similar to mine. > I am fairly new to Kafka land and probably there might be another way to fix > issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6975) AdminClient.deleteRecords() may cause replicas unable to fetch from beginning
[ https://issues.apache.org/jira/browse/KAFKA-6975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anna Povzner updated KAFKA-6975: Affects Version/s: 1.0.1 > AdminClient.deleteRecords() may cause replicas unable to fetch from beginning > - > > Key: KAFKA-6975 > URL: https://issues.apache.org/jira/browse/KAFKA-6975 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.1.0, 1.0.1 >Reporter: Anna Povzner >Assignee: Anna Povzner >Priority: Blocker > Fix For: 2.0.0, 1.0.2, 1.1.1 > > > AdminClient.deleteRecords(beforeOffset(offset)) will set log start offset to > the requested offset. If the requested offset is in the middle of the batch, > the replica will not be able to fetch from that offset (because it is in the > middle of the batch). > One use-case where this could cause problems is replica re-assignment. > Suppose we have a topic partition with 3 initial replicas, and at some point > the user issues AdminClient.deleteRecords() for the offset that falls in the > middle of the batch. It now becomes log start offset for this topic > partition. Suppose at some later time, the user starts partition > re-assignment to 3 new replicas. The new replicas (followers) will start with > HW = 0, will try to fetch from 0, then get "out of order offset" because 0 < > log start offset (LSO); the follower will be able to reset offset to LSO of > the leader and fetch LSO; the leader will send a batch in response with base > offset stop the fetcher thread. The end result is that the new replicas will not be > able to start fetching unless LSO moves to an offset that is not in the > middle of the batch, and the re-assignment will be stuck for a possibly a > very log time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6975) AdminClient.deleteRecords() may cause replicas unable to fetch from beginning
[ https://issues.apache.org/jira/browse/KAFKA-6975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anna Povzner updated KAFKA-6975: Fix Version/s: 1.1.1 1.0.2 > AdminClient.deleteRecords() may cause replicas unable to fetch from beginning > - > > Key: KAFKA-6975 > URL: https://issues.apache.org/jira/browse/KAFKA-6975 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Anna Povzner >Assignee: Anna Povzner >Priority: Blocker > Fix For: 2.0.0, 1.0.2, 1.1.1 > > > AdminClient.deleteRecords(beforeOffset(offset)) will set log start offset to > the requested offset. If the requested offset is in the middle of the batch, > the replica will not be able to fetch from that offset (because it is in the > middle of the batch). > One use-case where this could cause problems is replica re-assignment. > Suppose we have a topic partition with 3 initial replicas, and at some point > the user issues AdminClient.deleteRecords() for the offset that falls in the > middle of the batch. It now becomes log start offset for this topic > partition. Suppose at some later time, the user starts partition > re-assignment to 3 new replicas. The new replicas (followers) will start with > HW = 0, will try to fetch from 0, then get "out of order offset" because 0 < > log start offset (LSO); the follower will be able to reset offset to LSO of > the leader and fetch LSO; the leader will send a batch in response with base > offset stop the fetcher thread. The end result is that the new replicas will not be > able to start fetching unless LSO moves to an offset that is not in the > middle of the batch, and the re-assignment will be stuck for a possibly a > very log time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7048) NPE when creating connector
[
https://issues.apache.org/jira/browse/KAFKA-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509880#comment-16509880
]
Chia-Ping Tsai commented on KAFKA-7048:
---
{quote}Can you show the stack trace involving the NPE ?
{quote}
{code:java}
2018-06-13 00:57:49,324 DEBUG [qtp154246471-167]
errors.ConnectExceptionMapper(39): Uncaught exception in REST call to
/connectors
java.lang.NullPointerException
at
org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:79)
at
org.apache.kafka.connect.runtime.WorkerConfigTransformer.transform(WorkerConfigTransformer.java:42)
at
org.apache.kafka.connect.runtime.distributed.ClusterConfigState.connectorConfig(ClusterConfigState.java:120)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:559)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:535)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:271)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:220)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
> NPE when creating connector
> ---
>
> Key: KAFKA-7048
> URL: https://issues.apache.org/jira/browse/KAFKA-7048
> Project: Kafka
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 2.0.0
>
>
> KAFKA-6886 introduced the ConfigTransformer to transform the given
> configuration data. ConfigTransformer#transform(Map) expect
> the passed config won't be null but DistributedHerder#putConnectorConfig call
> the #transform before updating the snapshot (see below). Hence, it cause the
> NPE.
> {code:java}
> // Note that we use the updated connector config despite the fact that we
> don't have an updated
> // snapshot yet. The existing task info should still be accurate.
> Map map = configState.connectorConfig(connName);
> ConnectorInfo info = new ConnectorInfo(connName, config,
> configState.tasks(connName),
> map == null ? null :
> connectorTypeForClass(map.get(ConnectorConfig.CONNECTOR_CLASS_CONFIG)));
> callback.onCompletion(null, new Created<>(!exists, info));
> return null;{code}
> We can add a null check to "configs" (see below) to resolve the NPE. It means
> we WON'T pass the null configs to configTransformer
> {code:java}
> public Map connectorConfig(String connector) {
> Map configs = connectorConfigs.get(connector);
> if (configTransformer != null) { // add a condition "configs != null"
> configs = configTransformer.transform(connector, configs);
> }
> return configs;
> }{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7048) NPE when creating connector
[
https://issues.apache.org/jira/browse/KAFKA-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509862#comment-16509862
]
Ted Yu commented on KAFKA-7048:
---
Can you show the stack trace involving the NPE ?
> NPE when creating connector
> ---
>
> Key: KAFKA-7048
> URL: https://issues.apache.org/jira/browse/KAFKA-7048
> Project: Kafka
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 2.0.0
>
>
> KAFKA-6886 introduced the ConfigTransformer to transform the given
> configuration data. ConfigTransformer#transform(Map) expect
> the passed config won't be null but DistributedHerder#putConnectorConfig call
> the #transform before updating the snapshot (see below). Hence, it cause the
> NPE.
> {code:java}
> // Note that we use the updated connector config despite the fact that we
> don't have an updated
> // snapshot yet. The existing task info should still be accurate.
> Map map = configState.connectorConfig(connName);
> ConnectorInfo info = new ConnectorInfo(connName, config,
> configState.tasks(connName),
> map == null ? null :
> connectorTypeForClass(map.get(ConnectorConfig.CONNECTOR_CLASS_CONFIG)));
> callback.onCompletion(null, new Created<>(!exists, info));
> return null;{code}
> We can add a null check to "configs" (see below) to resolve the NPE. It means
> we WON'T pass the null configs to configTransformer
> {code:java}
> public Map connectorConfig(String connector) {
> Map configs = connectorConfigs.get(connector);
> if (configTransformer != null) { // add a condition "configs != null"
> configs = configTransformer.transform(connector, configs);
> }
> return configs;
> }{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7048) NPE when creating connector
[
https://issues.apache.org/jira/browse/KAFKA-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated KAFKA-7048:
--
Summary: NPE when creating connector (was: NPE when creating >1 connectors)
> NPE when creating connector
> ---
>
> Key: KAFKA-7048
> URL: https://issues.apache.org/jira/browse/KAFKA-7048
> Project: Kafka
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 2.0.0
>
>
> KAFKA-6886 introduced the ConfigTransformer to transform the given
> configuration data. ConfigTransformer#transform(Map) expect
> the passed config won't be null but DistributedHerder#putConnectorConfig call
> the #transform before updating the snapshot (see below). Hence, it cause the
> NPE.
> {code:java}
> // Note that we use the updated connector config despite the fact that we
> don't have an updated
> // snapshot yet. The existing task info should still be accurate.
> Map map = configState.connectorConfig(connName);
> ConnectorInfo info = new ConnectorInfo(connName, config,
> configState.tasks(connName),
> map == null ? null :
> connectorTypeForClass(map.get(ConnectorConfig.CONNECTOR_CLASS_CONFIG)));
> callback.onCompletion(null, new Created<>(!exists, info));
> return null;{code}
> We can add a null check to "configs" (see below) to resolve the NPE. It means
> we WON'T pass the null configs to configTransformer
> {code:java}
> public Map connectorConfig(String connector) {
> Map configs = connectorConfigs.get(connector);
> if (configTransformer != null) { // add a condition "configs != null"
> configs = configTransformer.transform(connector, configs);
> }
> return configs;
> }{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7048) NPE when creating connector
[
https://issues.apache.org/jira/browse/KAFKA-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated KAFKA-7048:
--
Issue Type: Bug (was: Improvement)
> NPE when creating connector
> ---
>
> Key: KAFKA-7048
> URL: https://issues.apache.org/jira/browse/KAFKA-7048
> Project: Kafka
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 2.0.0
>
>
> KAFKA-6886 introduced the ConfigTransformer to transform the given
> configuration data. ConfigTransformer#transform(Map) expect
> the passed config won't be null but DistributedHerder#putConnectorConfig call
> the #transform before updating the snapshot (see below). Hence, it cause the
> NPE.
> {code:java}
> // Note that we use the updated connector config despite the fact that we
> don't have an updated
> // snapshot yet. The existing task info should still be accurate.
> Map map = configState.connectorConfig(connName);
> ConnectorInfo info = new ConnectorInfo(connName, config,
> configState.tasks(connName),
> map == null ? null :
> connectorTypeForClass(map.get(ConnectorConfig.CONNECTOR_CLASS_CONFIG)));
> callback.onCompletion(null, new Created<>(!exists, info));
> return null;{code}
> We can add a null check to "configs" (see below) to resolve the NPE. It means
> we WON'T pass the null configs to configTransformer
> {code:java}
> public Map connectorConfig(String connector) {
> Map configs = connectorConfigs.get(connector);
> if (configTransformer != null) { // add a condition "configs != null"
> configs = configTransformer.transform(connector, configs);
> }
> return configs;
> }{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7048) NPE when creating >1 connectors
Chia-Ping Tsai created KAFKA-7048:
-
Summary: NPE when creating >1 connectors
Key: KAFKA-7048
URL: https://issues.apache.org/jira/browse/KAFKA-7048
Project: Kafka
Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
Fix For: 2.0.0
KAFKA-6886 introduced the ConfigTransformer to transform the given
configuration data. ConfigTransformer#transform(Map) expect the
passed config won't be null but DistributedHerder#putConnectorConfig call the
#transform before updating the snapshot (see below). Hence, it cause the NPE.
{code:java}
// Note that we use the updated connector config despite the fact that we don't
have an updated
// snapshot yet. The existing task info should still be accurate.
Map map = configState.connectorConfig(connName);
ConnectorInfo info = new ConnectorInfo(connName, config,
configState.tasks(connName),
map == null ? null :
connectorTypeForClass(map.get(ConnectorConfig.CONNECTOR_CLASS_CONFIG)));
callback.onCompletion(null, new Created<>(!exists, info));
return null;{code}
We can add a null check to "configs" (see below) to resolve the NPE. It means
we WON'T pass the null configs to configTransformer
{code:java}
public Map connectorConfig(String connector) {
Map configs = connectorConfigs.get(connector);
if (configTransformer != null) { // add a condition "configs != null"
configs = configTransformer.transform(connector, configs);
}
return configs;
}{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7045) Consumer may not be able to consume all messages when down-conversion is required
[ https://issues.apache.org/jira/browse/KAFKA-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruvil Shah reassigned KAFKA-7045: --- Assignee: Dhruvil Shah > Consumer may not be able to consume all messages when down-conversion is > required > - > > Key: KAFKA-7045 > URL: https://issues.apache.org/jira/browse/KAFKA-7045 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 0.11.0.0, 0.11.0.1, 1.0.0, 0.11.0.2, 1.1.0, 2.0.0, 1.0.1 >Reporter: Dhruvil Shah >Assignee: Dhruvil Shah >Priority: Major > Fix For: 2.0.0 > > Attachments: log-cleaner-test.zip > > > When down-conversion is required, the consumer might fail consuming messages > under certain conditions. Couple such cases are outlined below: > (1) When consuming from a compacted topic, it is possible that the consumer > wants to fetch messages that fall in the middle of a batch but the messages > have been compacted by the cleaner. For example, let's say we have the > following two segments. The brackets indicate a single batch of messages and > the numbers within are the message offsets. > Segment #1: [0, 1, 2], [3, 4, 5], [6, 7, 8] > Segment #2: [9, 10, 11], [12, 13, 14] > If the cleaner were to come in now and clean up messages with offsets 7 and > 8, the segments would look like the following: > Segment #1: [0, 1, 2], [3, 4, 5], [6] > Segment #2: [9, 10, 11], [12, 13, 14] > A consumer attempting to fetch messages at offset 7 will start reading the > batch starting at offset 6. During down-conversion, we will drop the record > starting at 6 it is less than the current fetch start offset. However, there > are no messages in the log following offset 6. In such cases, we return the > `FileRecords` itself which would cause the consumer to throw an exception > because it does not understand the stored message format. > (2) When consuming from a topic with transactional messages, down-conversion > usually drops control batches because these do not exist in V0 and V1 message > formats. If there are no message batches following the control batch in the > particular segment (or if we are at the end of the log), we would again get > no records after down-conversion and will return the `FileRecords`. Because > the consumer is not able to interpret control batches, it will again throw an > exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7045) Consumer may not be able to consume all messages when down-conversion is required
[ https://issues.apache.org/jira/browse/KAFKA-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruvil Shah updated KAFKA-7045: Fix Version/s: (was: 2.1.0) 2.0.0 > Consumer may not be able to consume all messages when down-conversion is > required > - > > Key: KAFKA-7045 > URL: https://issues.apache.org/jira/browse/KAFKA-7045 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 0.11.0.0, 0.11.0.1, 1.0.0, 0.11.0.2, 1.1.0, 2.0.0, 1.0.1 >Reporter: Dhruvil Shah >Priority: Major > Fix For: 2.0.0 > > Attachments: log-cleaner-test.zip > > > When down-conversion is required, the consumer might fail consuming messages > under certain conditions. Couple such cases are outlined below: > (1) When consuming from a compacted topic, it is possible that the consumer > wants to fetch messages that fall in the middle of a batch but the messages > have been compacted by the cleaner. For example, let's say we have the > following two segments. The brackets indicate a single batch of messages and > the numbers within are the message offsets. > Segment #1: [0, 1, 2], [3, 4, 5], [6, 7, 8] > Segment #2: [9, 10, 11], [12, 13, 14] > If the cleaner were to come in now and clean up messages with offsets 7 and > 8, the segments would look like the following: > Segment #1: [0, 1, 2], [3, 4, 5], [6] > Segment #2: [9, 10, 11], [12, 13, 14] > A consumer attempting to fetch messages at offset 7 will start reading the > batch starting at offset 6. During down-conversion, we will drop the record > starting at 6 it is less than the current fetch start offset. However, there > are no messages in the log following offset 6. In such cases, we return the > `FileRecords` itself which would cause the consumer to throw an exception > because it does not understand the stored message format. > (2) When consuming from a topic with transactional messages, down-conversion > usually drops control batches because these do not exist in V0 and V1 message > formats. If there are no message batches following the control batch in the > particular segment (or if we are at the end of the log), we would again get > no records after down-conversion and will return the `FileRecords`. Because > the consumer is not able to interpret control batches, it will again throw an > exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7047) Connect isolation whitelist does not include SimpleHeaderConverter
Randall Hauch created KAFKA-7047: Summary: Connect isolation whitelist does not include SimpleHeaderConverter Key: KAFKA-7047 URL: https://issues.apache.org/jira/browse/KAFKA-7047 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 1.1.0 Reporter: Randall Hauch Assignee: Randall Hauch The SimpleHeaderConverter added in 1.1.0 was never added to the PluginUtils whitelist so that this header converter is loaded in isolation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-2550) [Kafka][0.8.2.1][Performance]When there are a lot of partition under a Topic, there are serious performance degradation.
[
https://issues.apache.org/jira/browse/KAFKA-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-2550.
--
Resolution: Auto Closed
{color:#00}Closing inactive issue. Old clients are deprecated. Please
reopen if you think the issue still exists in newer versions.{color}
> [Kafka][0.8.2.1][Performance]When there are a lot of partition under a Topic,
> there are serious performance degradation.
>
>
> Key: KAFKA-2550
> URL: https://issues.apache.org/jira/browse/KAFKA-2550
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, producer
>Affects Versions: 0.8.2.1
>Reporter: yanwei
>Assignee: Neha Narkhede
>Priority: Major
>
> Because of business need to create a large number of partitions,I test the
> partition number of support.
> But I find When there are a lot of partition under a Topic, there are serious
> performance degradation.
> Through the analysis, in addition to the hard disk is bottleneck, the client
> is the bottleneck
> I use JProfile,producer and consumer 100 message(msg size:500byte)
> 1、Consumer high level API：(I find i can't upload picture?)
> ZookeeperConsumerConnector.scala-->rebalance
> -->val assignmentContext = new AssignmentContext(group, consumerIdString,
> config.excludeInternalTopics, zkClient)
> -->ZkUtils.getPartitionsForTopics(zkClient, myTopicThreadIds.keySet.toSeq)
> -->getPartitionAssignmentForTopics
> -->Json.parseFull(jsonPartitionMap)
> 1) one topic 400 partion：
> JProfile:48.6% cpu run time
> 2) ont topic 3000 partion:
> JProfile:97.8% cpu run time
> Maybe the file(jsonPartitionMap) is very big lead to parse is very slow.
> But this function is executed only once, so the problem should not be too
> big.
> 2、Producer Scala API：
> BrokerPartitionInfo.scala--->getBrokerPartitionInfo:
> partitionMetadata.map { m =>
> m.leader match {
> case Some(leader) =>
> //y00163442 delete log print
> debug("Partition [%s,%d] has leader %d".format(topic,
> m.partitionId, leader.id))
> new PartitionAndLeader(topic, m.partitionId, Some(leader.id))
> case None =>
> //y00163442 delete log print
> //debug("Partition [%s,%d] does not have a leader
> yet".format(topic, m.partitionId))
> new PartitionAndLeader(topic, m.partitionId, None)
> }
> }.sortWith((s, t) => s.partitionId < t.partitionId)
>
> When partitions number>25,the function 'format' cpu run time is 44.8%.
> Nearly half of the time consumption in the format function.whether the
> log print open, this format will be executed.Led to the decrease of the TPS
> for five times(25000--->5000).
>
> 3、Producer JAVA client（clients module）：
> function:org.apache.kafka.clients.producer.KafkaProducer.send
> I find the function 'send' cpu run time rise with the rising number of
> partitions ,when partions is 5000,the cpu run time is 60.8.
> Because Kafka broker side of CPU, memory, disk, the network didn't
> reach the bottleneck , No matter request.required.acks is set to 0 or 1, the
> results are similar, I doubt the send there may be some bottlenecks.
>
> Very unfortunately to upload pictures don't succeed, can't see the results.
> My test results, for a single server, a single hard disk can support 1000
> partitions, 7 hard disk can support 3000 partitions.If can solve the
> bottleneck for the client, then seven hard disk I estimate that can support
> more partitions.
> Actual production configuration, could be more partitions configuration under
> more than one TOPIC,Things could be better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-1181) Consolidate brokerList and topicPartitionInfo in BrokerPartitionInfo
[ https://issues.apache.org/jira/browse/KAFKA-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-1181. -- Resolution: Auto Closed Closing this as BrokerPartitionInfo class removed in KAFKA-6921 > Consolidate brokerList and topicPartitionInfo in BrokerPartitionInfo > > > Key: KAFKA-1181 > URL: https://issues.apache.org/jira/browse/KAFKA-1181 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Priority: Major > > brokerList in BrokerConfig is used to send the TopicMetadataRequest to known > brokers, and the broker id always starts at 0 and increase incrementally, AND > it is never updated in BrokerPartitionInfo even after the topic metadata > response has been received. > The real broker ids info is actually stored in topicPartitionInfo: > HashMap[String, TopicMetadata]. Which is refreshed with topic metadata > response. Therefore we could see different broker ids from logging entris > reporting failues of metadata request and failures of produce requests. > The solution here is to consolidate these two, reading the initial broker > list but keep it refreshed with topic metadata responses. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6761) Reduce Kafka Streams Footprint
[ https://issues.apache.org/jira/browse/KAFKA-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509742#comment-16509742 ] ASF GitHub Bot commented on KAFKA-6761: --- bbejeck opened a new pull request #5201: KAFKA-6761: Construct Physical Plan using Graph, Reduce streams footprint part III URL: https://github.com/apache/kafka/pull/5201 Sorry for the massive PR, but at this point it's very difficult to break up into smaller parts now that we are building the logical and physical plan. It's worth noting at them moment this PR does not include optimizations, for two reasons 1. We need a solution to providing properties for the topology build process. 2. Some of the repartition optimizations need more work The specific changes in this PR from the second PR include 1. Moved all graph objects into a separate package. This requires the graph objects to have public access but after looking at the code for some time it's worth the trade-off to have a cleaner code-base 2. Changed the types of graph nodes to names conveying more context 2. Build the entire physical plan from the graph, after `StreamsBuilder.build()` is called. 3. I'm currently working on applying optimizations and they will be applied soon. Those optimizations will include: 1. Re-using source topics as changelogs for KTables 2. Re-using sink topics as changelogs for statestore that output directly to sinks 3. Automatically repartition for key-changing operations followed by other DSL operations that would repartition on their own, reducing repartitions from N to 1. Other changes are addressed directly as review comments on the PR. Testing consists of using all existing streams tests to validate building the physical plan with graph ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Reduce Kafka Streams Footprint > -- > > Key: KAFKA-6761 > URL: https://issues.apache.org/jira/browse/KAFKA-6761 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bill Bejeck >Assignee: Bill Bejeck >Priority: Major > Fix For: 2.1.0 > > > The persistent storage footprint of a Kafka Streams application contains the > following aspects: > # The internal topics created on the Kafka cluster side. > # The materialized state stores on the Kafka Streams application instances > side. > There have been some questions about reducing these footprints, especially > since many of them are not necessary. For example, there are redundant > internal topics, as well as unnecessary state stores that takes up space but > also affect performance. When people are pushing Streams to production with > high traffic, this issue would be more common and severe. Reducing the > footprint of Streams have clear benefits for reducing resource utilization of > Kafka Streams applications, and also not creating pressure on broker's > capacities. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7038) Support AdminClient Example
[ https://issues.apache.org/jira/browse/KAFKA-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509727#comment-16509727 ] ASF GitHub Bot commented on KAFKA-7038: --- darionyaphet opened a new pull request #5200: [KAFKA-7038] Support AdminClient Example URL: https://github.com/apache/kafka/pull/5200 Add AdminClient Example include `describeCluster` , `createTopics` , `listTopics` , `describeTopics` and `deleteTopics` . ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support AdminClient Example > --- > > Key: KAFKA-7038 > URL: https://issues.apache.org/jira/browse/KAFKA-7038 > Project: Kafka > Issue Type: New Feature > Components: admin >Reporter: darion yaphet >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-3379) Update tools relying on old producer to use new producer.
[ https://issues.apache.org/jira/browse/KAFKA-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3379. -- Resolution: Fixed Fix Version/s: 2.0.0 > Update tools relying on old producer to use new producer. > - > > Key: KAFKA-3379 > URL: https://issues.apache.org/jira/browse/KAFKA-3379 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.10.0.0 >Reporter: Ashish Singh >Assignee: Ashish Singh >Priority: Major > Fix For: 2.0.0 > > > Following tools are using old producer. > * ReplicationVerificationTool > * SimpleConsumerShell > * GetOffsetShell > Old producer is being marked as deprecated in 0.10. These tools should be > updated to use new producer. To make sure that this update does not break > existing behavior. Below is the action plan. > For each tool that uses old producer. > * Add ducktape test to establish current behavior. > * Once the tests are committed and run fine, add patch for modification of > these tools. The ducktape tests added in previous step should confirm that > existing behavior is still intact. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7046) Support new Admin API for single topic
darion yaphet created KAFKA-7046: Summary: Support new Admin API for single topic Key: KAFKA-7046 URL: https://issues.apache.org/jira/browse/KAFKA-7046 Project: Kafka Issue Type: New Feature Components: admin Affects Versions: 1.1.0 Reporter: darion yaphet When I create topic delete and describe topic with AdminClient often use just one topic . Currently I must warp it into a collection . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-2572) zk connection instability
[
https://issues.apache.org/jira/browse/KAFKA-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-2572.
--
Resolution: Auto Closed
Closing inactive issue. Please reopen if the issue still exists in newer
versions.
> zk connection instability
> -
>
> Key: KAFKA-2572
> URL: https://issues.apache.org/jira/browse/KAFKA-2572
> Project: Kafka
> Issue Type: Bug
> Components: zkclient
>Affects Versions: 0.8.2.1
> Environment: zk version 3.4.6,
> CentOS 6, 2.6.32-504.1.3.el6.x86_64
>Reporter: John Firth
>Priority: Major
> Attachments: 090815-digest.log, 090815-full.log, 091115-digest.log,
> 091115-full.log.zip
>
>
> On several occasions, we've seen our process enter a cycle of: zk session
> expiry; new session creation; rebalancing activity; pause during which
> nothing is heard from the zk server. Sometimes, the reconnections are
> successful, elements are pulled from Kafka, but then disconnection and
> reconnection occurs shortly thereafter, causing OOMs when new elements are
> pulled in (although OOMs were not seen in the two cases attached as
> examples). Restarting the process that uses the zk client resolved the
> problems in both cases.
> This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show
> all logs entries minus entries particular to our application. For 09/08, the
> time span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00;
> for 11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to
> 2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining
> only error and warning entries, and entries containing any of: "begin
> rebalancing", "end rebalancing", "timed", and "zookeeper state". For the
> 09/11 digest logs, entries from the kafka.network.Processor logger are also
> excised for clarity. Unfortunately, debug logging was not enabled during
> these events.
> The 09/11 case shows repeated cycles of session expiry, followed by
> rebalancing activity, followed by a pause during which nothing is heard from
> the zk server, followed by a session timeout. A stable session seems to have
> been established at 2015-09-11T04:13:47.140-04:00, but messages of the form
> "I wrote this conflicted ephemeral node
> [{"version":1,"subscription":{"binlogs_mailchimp_us2":100},"pattern":"static","timestamp":"1441959227564"}]
> at
> /consumers/prologue-second-stage_prod_us2/ids/prologue-second-stage_prod_us2_app01.c1.prologue.prod.atl01.rsglab.com-1441812334972-b967b718
> a while back in a different session, hence I will backoff for this node to
> be deleted by Zookeeper and retry" were logged out repeatedly until we
> restarted the process after 2015-09-11T07:44:47.124-04:00, which marks the
> final entry in the log.
> The 09/08 case is a little more straightforward than the 09/11 case, in that
> a stable session was not established prior to our restarting the process.
> It's perhaps also noteworthy that in the 09/08 case, two timeouts for the
> same session are seen during a single rebalance, at
> 2015-09-08T12:52:19.107-04:00 and 2015-09-08T12:52:31.639-04:00. The
> rebalance in question begins at 2015-09-08T12:52:06.667-04:00.
> The connection to ZK expires and is restablished multiple times before the
> process is killed after 2015-09-08T13:13:41.655-04:00, which marks the last
> entry in the logs for this day.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6801) Restrict Consumer to fetch data from secure port only, and deny from non-secure port.
[ https://issues.apache.org/jira/browse/KAFKA-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509528#comment-16509528 ] Manikumar commented on KAFKA-6801: -- For PLAINTEXT connections, the principal will be ANONYMOUS. You can create ACLs to deny access to ANONYMOUS user. > Restrict Consumer to fetch data from secure port only, and deny from > non-secure port. > - > > Key: KAFKA-6801 > URL: https://issues.apache.org/jira/browse/KAFKA-6801 > Project: Kafka > Issue Type: Task > Components: admin, config, consumer, security >Affects Versions: 0.10.2.1 >Reporter: VinayKumar >Priority: Major > > I have listeners configured with 2 ports as below: (9092 -> Plaintext, 9092 > -> SASL_PLAIN) > listeners=PLAINTEXT://:9092, SASL_PLAIN://:9093 > For a topic, I want restrict Consumers to consume data from 9093 port only, > and consuming data from 9092 port should be denied. > I've gone through ACL concept, but haven't seen an option to restrict > Consumer pulling data from non-secure port (in this case- 9092) > Can someone please let me know if this is configurable ? > Can my requirement be fulfilled. Please provide necessary info. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-4690) IllegalStateException using DeleteTopicsRequest when delete.topic.enable=false
[
https://issues.apache.org/jira/browse/KAFKA-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar reassigned KAFKA-4690:
Assignee: Manikumar
> IllegalStateException using DeleteTopicsRequest when delete.topic.enable=false
> --
>
> Key: KAFKA-4690
> URL: https://issues.apache.org/jira/browse/KAFKA-4690
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.2.0
> Environment: OS X
>Reporter: Jon Chiu
>Assignee: Manikumar
>Priority: Major
> Attachments: delete-topics-request.java
>
>
> There is no indication as to why the delete request fails. Perhaps an error
> code?
> This can be reproduced with the following steps:
> 1. Start ZK and 1 broker (with default {{delete.topic.enable=false}})
> 2. Create a topic test
> {noformat}
> bin/kafka-topics.sh --zookeeper localhost:2181 \
> --create --topic test --partition 1 --replication-factor 1
> {noformat}
> 3. Delete topic by sending a DeleteTopicsRequest
> 4. An error is returned
> {noformat}
> org.apache.kafka.common.errors.DisconnectException
> {noformat}
> or
> {noformat}
> java.lang.IllegalStateException: Attempt to retrieve exception from future
> which hasn't failed
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.exception(RequestFuture.java:99)
> at
> io.confluent.adminclient.KafkaAdminClient.send(KafkaAdminClient.java:195)
> at
> io.confluent.adminclient.KafkaAdminClient.deleteTopic(KafkaAdminClient.java:152)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6631) Kafka Streams - Rebalancing exception in Kafka 1.0.0
[
https://issues.apache.org/jira/browse/KAFKA-6631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509297#comment-16509297
]
Ashish Surana edited comment on KAFKA-6631 at 6/12/18 10:07 AM:
{code:java}
RecordTooLargeException{code}
This issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size depends on number of consumer threads, and number of tasks
(active + replica). We have 512 partitions, 256 threads and 1 replica, so
entire assignment data size is around 12MB for us.
I think it would be a good idea to add this into documentation somewhere that
assignment data size is indirectly limited by these message settings on the
broker side.
was (Author: asurana):
{code:java}
RecordTooLargeException{code}
This issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We have 512 partitions and 1
replica, so entire assignment data size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size or total tasks (active tasks + replica tasks) is
indirectly limited by these message settings on the broker side.
> Kafka Streams - Rebalancing exception in Kafka 1.0.0
>
>
> Key: KAFKA-6631
> URL: https://issues.apache.org/jira/browse/KAFKA-6631
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
> Environment: Container Linux by CoreOS 1576.5.0
>Reporter: Alexander Ivanichev
>Priority: Critical
>
>
> In Kafka Streams 1.0.0, we saw a strange rebalance error, our stream app
> performs window based aggregations, sometimes on start when all stream
> workers join the app just crash, however if we enable only one worker than
> it works fine, sometime 2 workers work just fine, but when third join the app
> crashes again, some critical issue with rebalance.
> {code:java}
> 018-03-08T18:51:01.226243000Z org.apache.kafka.common.KafkaException:
> Unexpected error from SyncGroup: The server experienced an unexpected error
> when processing the request
> 2018-03-08T18:51:01.226557000Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:566)
> 2018-03-08T18:51:01.22686Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:539)
> 2018-03-08T18:51:01.227328000Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:808)
> 2018-03-08T18:51:01.22763Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:788)
> 2018-03-08T18:51:01.228152000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
> 2018-03-08T18:51:01.228449000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
> 2018-03-08T18:51:01.228897000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
> 2018-03-08T18:51:01.229196000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:506)
> 2018-03-08T18:51:01.229673000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)
> 2018-03-08T18:51:01.229971000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:268)
> 2018-03-08T18:51:01.230436000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
> 2018-03-08T18:51:01.230749000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:174)
> 2018-03-08T18:51:01.231065000Z at
>
[jira] [Comment Edited] (KAFKA-6976) Kafka Streams instances going in to DEAD state
[
https://issues.apache.org/jira/browse/KAFKA-6976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509296#comment-16509296
]
Ashish Surana edited comment on KAFKA-6976 at 6/12/18 10:07 AM:
{code:java}
RecordTooLargeException{code}
This issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size depends on number of consumer threads, and number of tasks
(active + replica). We have 512 partitions, 256 threads and 1 replica, so
entire assignment data size is around 12MB for us.
I think it would be a good idea to add this into documentation somewhere that
assignment data size is indirectly limited by these message settings on the
broker side.
was (Author: asurana):
{code:java}
RecordTooLargeException{code}
This issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We have 512 partitions and 1
replica, so entire assignment data size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size or total tasks (active tasks + replica tasks) is
indirectly limited by these message settings on the broker side.
> Kafka Streams instances going in to DEAD state
> --
>
> Key: KAFKA-6976
> URL: https://issues.apache.org/jira/browse/KAFKA-6976
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Deepak Goyal
>Priority: Blocker
> Attachments: kafkaStreamsDeadState.log
>
>
> We are using Kafka 0.10.2.0, Kafka-Streams 1.1.0. We have Kafka Cluster of 16
> machines, and topic that is being consumed by Kafka Streams has 256
> partitions. We spawned 400 machines of Kakfa Streams application. We see that
> all of the StreamThreads go in to DEAD state.
> {quote}{{[2018-05-25 05:59:29,282] INFO stream-thread
> [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1] State transition
> from PENDING_SHUTDOWN to DEAD
> (org.apache.kafka.streams.processor.internals.StreamThread) [2018-05-25
> 05:59:29,282] INFO stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7]
> State transition from REBALANCING to ERROR
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] WARN
> stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] INFO
> stream-thread [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1]
> Shutdown complete
> (org.apache.kafka.streams.processor.internals.StreamThread)}}
> {quote}
> Please note that when we only have 100 kafka-streams application machines,
> things are working as expected. We see that instances are consuming messages
> from topic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-6351) libs directory has duplicate javassist jars
[ https://issues.apache.org/jira/browse/KAFKA-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6351. -- Resolution: Fixed Fix Version/s: 2.0.0 Closing this as javassist jar is resolving to single version in latest code. > libs directory has duplicate javassist jars > --- > > Key: KAFKA-6351 > URL: https://issues.apache.org/jira/browse/KAFKA-6351 > Project: Kafka > Issue Type: Bug > Components: build >Affects Versions: 1.0.0 >Reporter: pre sto >Priority: Minor > Fix For: 2.0.0 > > > Downloaded kafka_2.11-1.0.0 and noticed duplicate jars under libs > javassist-3.20.0-GA.jar > javassist-3.21.0-GA.jar > I assume that's a mistake -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6351) libs directory has duplicate javassist jars
[
https://issues.apache.org/jira/browse/KAFKA-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509401#comment-16509401
]
ASF GitHub Bot commented on KAFKA-6351:
---
omkreddy closed pull request #4719: KAFKA-6351: Prevent copying javassist
library to Kafka distribution from tools project
URL: https://github.com/apache/kafka/pull/4719
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/build.gradle b/build.gradle
index 5e4c35643c2..1683ed692b6 100644
--- a/build.gradle
+++ b/build.gradle
@@ -721,7 +721,7 @@ project(':core') {
from(configurations.archives.artifacts.files) { into("libs/") }
from(project.siteDocsTar) { into("site-docs/") }
from(project(':tools').jar) { into("libs/") }
-from(project(':tools').configurations.runtime) { into("libs/") }
+from(project(':tools').configurations.runtime) { into("libs/") exclude
'**/*javassist*'}
from(project(':connect:api').jar) { into("libs/") }
from(project(':connect:api').configurations.runtime) { into("libs/") }
from(project(':connect:runtime').jar) { into("libs/") }
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> libs directory has duplicate javassist jars
> ---
>
> Key: KAFKA-6351
> URL: https://issues.apache.org/jira/browse/KAFKA-6351
> Project: Kafka
> Issue Type: Bug
> Components: build
>Affects Versions: 1.0.0
>Reporter: pre sto
>Priority: Minor
>
> Downloaded kafka_2.11-1.0.0 and noticed duplicate jars under libs
> javassist-3.20.0-GA.jar
> javassist-3.21.0-GA.jar
> I assume that's a mistake
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6914) Kafka Connect - Plugins class should have a constructor that can take in parent ClassLoader
[ https://issues.apache.org/jira/browse/KAFKA-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509378#comment-16509378 ] Frank Lyaruu commented on KAFKA-6914: - I have a similar issue (not with Spring but by embedding it into an OSGi bundle). My specific problem is that the DelegatingClassLoader does not resolve using its own classloader. Plugins.java:61 creates a DelegatingClassLoader, but it uses the system classloader as a parent, which won't resolve anything in my case. If I replace the ClassLoader.getSystemClassLoader() by DelegatingClassLoader.class.getClassLoader() in DelegatingClassLoader.java:98 it works for me. Isn't that a more sensible fallback classloader than the system classloader? Or is there an intention I'm not aware of? > Kafka Connect - Plugins class should have a constructor that can take in > parent ClassLoader > --- > > Key: KAFKA-6914 > URL: https://issues.apache.org/jira/browse/KAFKA-6914 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Sriram KS >Priority: Minor > Fix For: 1.1.1 > > > Currently Plugins class has a single constructor that takes in map of props. > Please make Plugin class to have a constructor that takes in a classLoader as > well and use it to set DelegationClassLoader's parent classLoader. > Reason: > This will be useful if i am already having a managed class Loader environment > like a Spring boot app which resolves my class dependencies using my > maven/gradle dependency management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6562) KIP-255: OAuth Authentication via SASL/OAUTHBEARER
[
https://issues.apache.org/jira/browse/KAFKA-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509370#comment-16509370
]
ASF GitHub Bot commented on KAFKA-6562:
---
ijuma closed pull request #5110: KAFKA-6562: (follow-up) Publish
"jackson-databind" lib as provided scope dependency in clients maven artifact
URL: https://github.com/apache/kafka/pull/5110
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/build.gradle b/build.gradle
index 911ba09edec..60476eebcf1 100644
--- a/build.gradle
+++ b/build.gradle
@@ -785,11 +785,20 @@ project(':examples') {
project(':clients') {
archivesBaseName = "kafka-clients"
+ configurations {
+jacksonDatabindConfig
+ }
+
+ // add jacksonDatabindConfig as provided scope config with high priority
(1000)
+ conf2ScopeMappings.addMapping(1000, configurations.jacksonDatabindConfig,
"provided")
+
dependencies {
compile libs.lz4
compile libs.snappy
compile libs.slf4jApi
-compile libs.jacksonDatabind // for SASL/OAUTHBEARER bearer token parsing
+compileOnly libs.jacksonDatabind // for SASL/OAUTHBEARER bearer token
parsing
+
+jacksonDatabindConfig libs.jacksonDatabind // to publish as provided scope
dependency.
testCompile libs.bcpkix
testCompile libs.junit
@@ -798,6 +807,7 @@ project(':clients') {
testCompile libs.powermockEasymock
testRuntime libs.slf4jlog4j
+testRuntime libs.jacksonDatabind
}
task determineCommitId {
diff --git a/docs/security.html b/docs/security.html
index 877ca579a52..c1682ff8fbf 100644
--- a/docs/security.html
+++ b/docs/security.html
@@ -715,6 +715,8 @@ 7.3
Authentication using SASL
security.protocol=SASL_SSL (or SASL_PLAINTEXT if non-production)
sasl.mechanism=OAUTHBEARER
+ The default implementation of SASL/OAUTHBEARER depends on the
jackson-databind library.
+ Since it's an optional dependency, users have to configure it
as a dependency via their build tool.
Unsecured Token Creation
Options for SASL/OAUTHBEARER
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> KIP-255: OAuth Authentication via SASL/OAUTHBEARER
> --
>
> Key: KAFKA-6562
> URL: https://issues.apache.org/jira/browse/KAFKA-6562
> Project: Kafka
> Issue Type: Improvement
> Components: clients
>Reporter: Ron Dagostino
>Assignee: Ron Dagostino
>Priority: Major
> Fix For: 2.0.0
>
>
> KIP-255: OAuth Authentication via SASL/OAUTHBEARER
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75968876)
> proposes adding the ability to authenticate to Kafka with OAuth 2 bearer
> tokens using the OAUTHBEARER SASL mechanism. Token retrieval and token
> validation are both pluggable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-7029) ReplicaVerificationTool should not use the deprecated SimpleConsumer
[ https://issues.apache.org/jira/browse/KAFKA-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-7029. Resolution: Fixed > ReplicaVerificationTool should not use the deprecated SimpleConsumer > > > Key: KAFKA-7029 > URL: https://issues.apache.org/jira/browse/KAFKA-7029 > Project: Kafka > Issue Type: Bug >Reporter: Ismael Juma >Assignee: Manikumar >Priority: Major > Fix For: 2.0.0 > > > Unless there's a reason not to, the simplest would be to use KafkaConsumer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7029) ReplicaVerificationTool should not use the deprecated SimpleConsumer
[ https://issues.apache.org/jira/browse/KAFKA-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-7029: --- Fix Version/s: 2.0.0 > ReplicaVerificationTool should not use the deprecated SimpleConsumer > > > Key: KAFKA-7029 > URL: https://issues.apache.org/jira/browse/KAFKA-7029 > Project: Kafka > Issue Type: Bug >Reporter: Ismael Juma >Assignee: Manikumar >Priority: Major > Fix For: 2.0.0 > > > Unless there's a reason not to, the simplest would be to use KafkaConsumer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6976) Kafka Streams instances going in to DEAD state
[
https://issues.apache.org/jira/browse/KAFKA-6976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509296#comment-16509296
]
Ashish Surana edited comment on KAFKA-6976 at 6/12/18 7:48 AM:
---
{code:java}
RecordTooLargeException{code}
This issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We have 512 partitions and 1
replica, so entire assignment data size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size or total tasks (active tasks + replica tasks) is
indirectly limited by these message settings on the broker side.
was (Author: asurana):
The issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We have 512 partitions and 1
replica, so entire assignment data size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size or total tasks (active tasks + replica tasks) is
indirectly limited by these message settings on the broker side.
> Kafka Streams instances going in to DEAD state
> --
>
> Key: KAFKA-6976
> URL: https://issues.apache.org/jira/browse/KAFKA-6976
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Deepak Goyal
>Priority: Blocker
> Attachments: kafkaStreamsDeadState.log
>
>
> We are using Kafka 0.10.2.0, Kafka-Streams 1.1.0. We have Kafka Cluster of 16
> machines, and topic that is being consumed by Kafka Streams has 256
> partitions. We spawned 400 machines of Kakfa Streams application. We see that
> all of the StreamThreads go in to DEAD state.
> {quote}{{[2018-05-25 05:59:29,282] INFO stream-thread
> [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1] State transition
> from PENDING_SHUTDOWN to DEAD
> (org.apache.kafka.streams.processor.internals.StreamThread) [2018-05-25
> 05:59:29,282] INFO stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7]
> State transition from REBALANCING to ERROR
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] WARN
> stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] INFO
> stream-thread [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1]
> Shutdown complete
> (org.apache.kafka.streams.processor.internals.StreamThread)}}
> {quote}
> Please note that when we only have 100 kafka-streams application machines,
> things are working as expected. We see that instances are consuming messages
> from topic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6631) Kafka Streams - Rebalancing exception in Kafka 1.0.0
[
https://issues.apache.org/jira/browse/KAFKA-6631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509297#comment-16509297
]
Ashish Surana edited comment on KAFKA-6631 at 6/12/18 7:48 AM:
---
{code:java}
RecordTooLargeException{code}
This issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We have 512 partitions and 1
replica, so entire assignment data size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size or total tasks (active tasks + replica tasks) is
indirectly limited by these message settings on the broker side.
was (Author: asurana):
The issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We have 512 partitions and 1
replica, so entire assignment data size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size or total tasks (active tasks + replica tasks) is
indirectly limited by these message settings on the broker side.
> Kafka Streams - Rebalancing exception in Kafka 1.0.0
>
>
> Key: KAFKA-6631
> URL: https://issues.apache.org/jira/browse/KAFKA-6631
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
> Environment: Container Linux by CoreOS 1576.5.0
>Reporter: Alexander Ivanichev
>Priority: Critical
>
>
> In Kafka Streams 1.0.0, we saw a strange rebalance error, our stream app
> performs window based aggregations, sometimes on start when all stream
> workers join the app just crash, however if we enable only one worker than
> it works fine, sometime 2 workers work just fine, but when third join the app
> crashes again, some critical issue with rebalance.
> {code:java}
> 018-03-08T18:51:01.226243000Z org.apache.kafka.common.KafkaException:
> Unexpected error from SyncGroup: The server experienced an unexpected error
> when processing the request
> 2018-03-08T18:51:01.226557000Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:566)
> 2018-03-08T18:51:01.22686Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:539)
> 2018-03-08T18:51:01.227328000Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:808)
> 2018-03-08T18:51:01.22763Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:788)
> 2018-03-08T18:51:01.228152000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
> 2018-03-08T18:51:01.228449000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
> 2018-03-08T18:51:01.228897000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
> 2018-03-08T18:51:01.229196000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:506)
> 2018-03-08T18:51:01.229673000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)
> 2018-03-08T18:51:01.229971000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:268)
> 2018-03-08T18:51:01.230436000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
> 2018-03-08T18:51:01.230749000Z at
>
[jira] [Comment Edited] (KAFKA-6631) Kafka Streams - Rebalancing exception in Kafka 1.0.0
[
https://issues.apache.org/jira/browse/KAFKA-6631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509297#comment-16509297
]
Ashish Surana edited comment on KAFKA-6631 at 6/12/18 7:47 AM:
---
The issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We have 512 partitions and 1
replica, so entire assignment data size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size or total tasks (active tasks + replica tasks) is
indirectly limited by these message settings on the broker side.
was (Author: asurana):
The issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We had 512 partitions and 1
replica, so entire assignment size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size (or active tasks + replica tasks) is indirectly limited by
these settings on the broker side.
> Kafka Streams - Rebalancing exception in Kafka 1.0.0
>
>
> Key: KAFKA-6631
> URL: https://issues.apache.org/jira/browse/KAFKA-6631
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
> Environment: Container Linux by CoreOS 1576.5.0
>Reporter: Alexander Ivanichev
>Priority: Critical
>
>
> In Kafka Streams 1.0.0, we saw a strange rebalance error, our stream app
> performs window based aggregations, sometimes on start when all stream
> workers join the app just crash, however if we enable only one worker than
> it works fine, sometime 2 workers work just fine, but when third join the app
> crashes again, some critical issue with rebalance.
> {code:java}
> 018-03-08T18:51:01.226243000Z org.apache.kafka.common.KafkaException:
> Unexpected error from SyncGroup: The server experienced an unexpected error
> when processing the request
> 2018-03-08T18:51:01.226557000Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:566)
> 2018-03-08T18:51:01.22686Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:539)
> 2018-03-08T18:51:01.227328000Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:808)
> 2018-03-08T18:51:01.22763Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:788)
> 2018-03-08T18:51:01.228152000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
> 2018-03-08T18:51:01.228449000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
> 2018-03-08T18:51:01.228897000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
> 2018-03-08T18:51:01.229196000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:506)
> 2018-03-08T18:51:01.229673000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)
> 2018-03-08T18:51:01.229971000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:268)
> 2018-03-08T18:51:01.230436000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
> 2018-03-08T18:51:01.230749000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:174)
> 2018-03-08T18:51:01.231065000Z at
>
[jira] [Comment Edited] (KAFKA-6976) Kafka Streams instances going in to DEAD state
[
https://issues.apache.org/jira/browse/KAFKA-6976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509296#comment-16509296
]
Ashish Surana edited comment on KAFKA-6976 at 6/12/18 7:47 AM:
---
The issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We have 512 partitions and 1
replica, so entire assignment data size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size or total tasks (active tasks + replica tasks) is
indirectly limited by these message settings on the broker side.
was (Author: asurana):
The issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We had 512 partitions and 1
replica, so entire assignment size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size (or active tasks + replica tasks) is indirectly limited by
these settings on the broker side.
> Kafka Streams instances going in to DEAD state
> --
>
> Key: KAFKA-6976
> URL: https://issues.apache.org/jira/browse/KAFKA-6976
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Deepak Goyal
>Priority: Blocker
> Attachments: kafkaStreamsDeadState.log
>
>
> We are using Kafka 0.10.2.0, Kafka-Streams 1.1.0. We have Kafka Cluster of 16
> machines, and topic that is being consumed by Kafka Streams has 256
> partitions. We spawned 400 machines of Kakfa Streams application. We see that
> all of the StreamThreads go in to DEAD state.
> {quote}{{[2018-05-25 05:59:29,282] INFO stream-thread
> [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1] State transition
> from PENDING_SHUTDOWN to DEAD
> (org.apache.kafka.streams.processor.internals.StreamThread) [2018-05-25
> 05:59:29,282] INFO stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7]
> State transition from REBALANCING to ERROR
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] WARN
> stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] INFO
> stream-thread [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1]
> Shutdown complete
> (org.apache.kafka.streams.processor.internals.StreamThread)}}
> {quote}
> Please note that when we only have 100 kafka-streams application machines,
> things are working as expected. We see that instances are consuming messages
> from topic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7029) ReplicaVerificationTool should not use the deprecated SimpleConsumer
[
https://issues.apache.org/jira/browse/KAFKA-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509302#comment-16509302
]
ASF GitHub Bot commented on KAFKA-7029:
---
ijuma closed pull request #5188: KAFKA-7029: Update ReplicaVerificationTool to
use Java Consumer
URL: https://github.com/apache/kafka/pull/5188
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/core/src/main/scala/kafka/server/ReplicaFetcherBlockingSend.scala
b/core/src/main/scala/kafka/server/ReplicaFetcherBlockingSend.scala
index 0bf2bd3c9ff..4c7adfbe9cd 100644
--- a/core/src/main/scala/kafka/server/ReplicaFetcherBlockingSend.scala
+++ b/core/src/main/scala/kafka/server/ReplicaFetcherBlockingSend.scala
@@ -86,7 +86,7 @@ class ReplicaFetcherBlockingSend(sourceBroker: BrokerEndPoint,
)
}
- override def sendRequest(requestBuilder: Builder[_ <: AbstractRequest]):
ClientResponse = {
+ override def sendRequest(requestBuilder: Builder[_ <: AbstractRequest]):
ClientResponse = {
try {
if (!NetworkClientUtils.awaitReady(networkClient, sourceNode, time,
socketTimeout))
throw new SocketTimeoutException(s"Failed to connect within
$socketTimeout ms")
diff --git a/core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala
b/core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala
index 0408e9212a3..b1e694688b7 100644
--- a/core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala
+++ b/core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala
@@ -17,25 +17,35 @@
package kafka.tools
+import java.net.SocketTimeoutException
import java.text.SimpleDateFormat
-import java.util.Date
+import java.util
import java.util.concurrent.CountDownLatch
-import java.util.concurrent.atomic.AtomicReference
+import java.util.concurrent.atomic.{AtomicInteger, AtomicReference}
import java.util.regex.{Pattern, PatternSyntaxException}
+import java.util.{Date, Properties}
import joptsimple.OptionParser
import kafka.api._
-import kafka.client.ClientUtils
-import kafka.cluster.BrokerEndPoint
-import kafka.common.TopicAndPartition
-import kafka.consumer.{ConsumerConfig, SimpleConsumer, Whitelist}
-import kafka.message.{ByteBufferMessageSet, MessageSet}
+import kafka.consumer.Whitelist
import kafka.utils._
-import org.apache.kafka.common.protocol.Errors
-import org.apache.kafka.common.utils.Time
+import org.apache.kafka.clients._
+import org.apache.kafka.clients.admin.{ListTopicsOptions, TopicDescription}
+import org.apache.kafka.clients.consumer.{ConsumerConfig, KafkaConsumer}
+import org.apache.kafka.common.metrics.Metrics
+import org.apache.kafka.common.network.{NetworkReceive, Selectable, Selector}
+import org.apache.kafka.common.protocol.{ApiKeys, Errors}
+import org.apache.kafka.common.record.MemoryRecords
+import org.apache.kafka.common.requests.AbstractRequest.Builder
+import org.apache.kafka.common.requests.{AbstractRequest, FetchResponse,
ListOffsetRequest, FetchRequest => JFetchRequest}
+import org.apache.kafka.common.serialization.StringDeserializer
+import org.apache.kafka.common.utils.{LogContext, Time}
+import org.apache.kafka.common.{Node, TopicPartition}
+
+import scala.collection.JavaConverters._
/**
- * For verifying the consistency among replicas.
+ * For verifying the consistency among replicas.
*
* 1. start a fetcher on every broker.
* 2. each fetcher does the following
@@ -44,11 +54,11 @@ import org.apache.kafka.common.utils.Time
*2.3 waits for all other fetchers to finish step 2.2
*2.4 one of the fetchers verifies the consistency of fetched results
among replicas
*
- * The consistency verification is up to the high watermark. The tool reports
the
- * max lag between the verified offset and the high watermark among all
partitions.
+ * The consistency verification is up to the high watermark. The tool reports
the
+ * max lag between the verified offset and the high watermark among all
partitions.
*
- * If a broker goes down, the verification of the partitions on that broker
is delayed
- * until the broker is up again.
+ * If a broker goes down, the verification of the partitions on that broker is
delayed
+ * until the broker is up again.
*
* Caveats:
* 1. The tools needs all brokers to be up at startup time.
@@ -56,7 +66,7 @@ import org.apache.kafka.common.utils.Time
*/
object ReplicaVerificationTool extends Logging {
- val clientId= "replicaVerificationTool"
+ val clientId = "replicaVerificationTool"
val dateFormatString = "-MM-dd HH:mm:ss,SSS"
val dateFormat = new SimpleDateFormat(dateFormatString)
@@ -74,7 +84,7 @@ object ReplicaVerificationTool extends Logging {
.withRequiredArg
[jira] [Commented] (KAFKA-6631) Kafka Streams - Rebalancing exception in Kafka 1.0.0
[
https://issues.apache.org/jira/browse/KAFKA-6631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509297#comment-16509297
]
Ashish Surana commented on KAFKA-6631:
--
The issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We had 512 partitions and 1
replica, so entire assignment size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size (or active tasks + replica tasks) is indirectly limited by
these settings on the broker side.
> Kafka Streams - Rebalancing exception in Kafka 1.0.0
>
>
> Key: KAFKA-6631
> URL: https://issues.apache.org/jira/browse/KAFKA-6631
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
> Environment: Container Linux by CoreOS 1576.5.0
>Reporter: Alexander Ivanichev
>Priority: Critical
>
>
> In Kafka Streams 1.0.0, we saw a strange rebalance error, our stream app
> performs window based aggregations, sometimes on start when all stream
> workers join the app just crash, however if we enable only one worker than
> it works fine, sometime 2 workers work just fine, but when third join the app
> crashes again, some critical issue with rebalance.
> {code:java}
> 018-03-08T18:51:01.226243000Z org.apache.kafka.common.KafkaException:
> Unexpected error from SyncGroup: The server experienced an unexpected error
> when processing the request
> 2018-03-08T18:51:01.226557000Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:566)
> 2018-03-08T18:51:01.22686Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:539)
> 2018-03-08T18:51:01.227328000Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:808)
> 2018-03-08T18:51:01.22763Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:788)
> 2018-03-08T18:51:01.228152000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
> 2018-03-08T18:51:01.228449000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
> 2018-03-08T18:51:01.228897000Z at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
> 2018-03-08T18:51:01.229196000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:506)
> 2018-03-08T18:51:01.229673000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)
> 2018-03-08T18:51:01.229971000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:268)
> 2018-03-08T18:51:01.230436000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
> 2018-03-08T18:51:01.230749000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:174)
> 2018-03-08T18:51:01.231065000Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:364)
> 2018-03-08T18:51:01.231584000Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
> 2018-03-08T18:51:01.231911000Z at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:295)
> 2018-03-08T18:51:01.23219Z at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1138)
> 2018-03-08T18:51:01.232643000Z at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1103)
> 2018-03-08T18:51:01.233121000Z at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:851)
> 2018-03-08T18:51:01.233409000Z at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:808)
> 2018-03-08T18:51:01.23372Z at
>
[jira] [Commented] (KAFKA-6976) Kafka Streams instances going in to DEAD state
[
https://issues.apache.org/jira/browse/KAFKA-6976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509296#comment-16509296
]
Ashish Surana commented on KAFKA-6976:
--
The issue is resolved after making some changes to broker settings:
*message.max.bytes=33554432*
*replica.fetch.max.bytes=33554432*
Here these are set to 32MB, but earlier these settings were set to 2MB for us.
Like you explained, broker saves this assignment as one single message in
__consumer_offset topics so the assignment size is also limited by these two
message settings on kafka broker.
Assignment size is directly proportional to number of tasks (we observed that
assignment roughly contains 11KB for each task). We had 512 partitions and 1
replica, so entire assignment size is 11KB * 512(partitions) * 2(1active +
1replica task) ~= 12MB.
I think it would be a good idea to add this into documentation somewhere that
assignment data size (or active tasks + replica tasks) is indirectly limited by
these settings on the broker side.
> Kafka Streams instances going in to DEAD state
> --
>
> Key: KAFKA-6976
> URL: https://issues.apache.org/jira/browse/KAFKA-6976
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Deepak Goyal
>Priority: Blocker
> Attachments: kafkaStreamsDeadState.log
>
>
> We are using Kafka 0.10.2.0, Kafka-Streams 1.1.0. We have Kafka Cluster of 16
> machines, and topic that is being consumed by Kafka Streams has 256
> partitions. We spawned 400 machines of Kakfa Streams application. We see that
> all of the StreamThreads go in to DEAD state.
> {quote}{{[2018-05-25 05:59:29,282] INFO stream-thread
> [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1] State transition
> from PENDING_SHUTDOWN to DEAD
> (org.apache.kafka.streams.processor.internals.StreamThread) [2018-05-25
> 05:59:29,282] INFO stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7]
> State transition from REBALANCING to ERROR
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] WARN
> stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7] All stream threads
> have died. The instance will be in error state and should be closed.
> (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] INFO
> stream-thread [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1]
> Shutdown complete
> (org.apache.kafka.streams.processor.internals.StreamThread)}}
> {quote}
> Please note that when we only have 100 kafka-streams application machines,
> things are working as expected. We see that instances are consuming messages
> from topic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6161) Introduce new serdes interfaces with empty configure() and close()
[
https://issues.apache.org/jira/browse/KAFKA-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509295#comment-16509295
]
Chia-Ping Tsai commented on KAFKA-6161:
---
I'm going to write the KIP. [~ewencp] Could I take over this jira?
> Introduce new serdes interfaces with empty configure() and close()
> --
>
> Key: KAFKA-6161
> URL: https://issues.apache.org/jira/browse/KAFKA-6161
> Project: Kafka
> Issue Type: Improvement
> Components: clients, streams
>Reporter: Evgeny Veretennikov
>Assignee: Evgeny Veretennikov
>Priority: Major
> Labels: needs-kip
>
> {{Serializer}}, {{Deserializer}} and {{Serde}} interfaces have methods
> {{configure()}} and {{close()}}. Pretty often one want to leave these methods
> empty. For example, a lot of serializers inside
> {{org.apache.kafka.common.serialization}} package have these methods empty:
> {code}
> @Override
> public void configure(Map configs, boolean isKey) {
> // nothing to do
> }
> @Override
> public void close() {
> // nothing to do
> }
> {code}
> To avoid such boilerplate, we may create new interfaces (like
> {{UnconfiguredSerializer}}), in which we will define these methods empty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6806) Unable to validate sink connectors without "topics" component which is not required
[
https://issues.apache.org/jira/browse/KAFKA-6806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509256#comment-16509256
]
Dong Lin commented on KAFKA-6806:
-
Hey [~rhauch], this issue has been marked as blocking issue for 1.1.1 but there
does not seem to be person actively working on this. Do you know who will be
working on this issue, or do we actually need to block 1.1.1 release until it
is fixed?
> Unable to validate sink connectors without "topics" component which is not
> required
> ---
>
> Key: KAFKA-6806
> URL: https://issues.apache.org/jira/browse/KAFKA-6806
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 1.1.0
> Environment: CP4.1., Centos7
>Reporter: Ivan Majnarić
>Priority: Blocker
> Fix For: 2.0.0, 1.1.1
>
>
> The bug is happening when you try to create new connector through for example
> kafka-connect-ui.
> While both source and sink connectors were able to be validated through REST
> without "topics" as add-on with "connector.class" like this:
> {code:java}
> PUT /
> http://connect-url:8083/connector-plugins/com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector/config/validate
> {
> "connector.class":
> "com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector",
> }{code}
> In the new version of CP4.1 you still can validate *source connectors* but
> not *sink connectors*. If you want to validate sink connectors you need to
> add to request -> "topics" config, like:
> {code:java}
> PUT /
> http://connect-url:8083/connector-plugins/com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector/config/validate
> {
> "connector.class":
> "com.datamountaineer.streamreactor.connect.cassandra.sink.CassandraSinkConnector",
> "topics": "test-topic"
> }{code}
> So there is a little missmatch of the ways how to validate connectors which I
> think happened accidentally.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
