[jira] [Created] (KAFKA-6050) Cannot alter default topic config
Tom Bentley created KAFKA-6050:
--
Summary: Cannot alter default topic config
Key: KAFKA-6050
URL: https://issues.apache.org/jira/browse/KAFKA-6050
Project: Kafka
Issue Type: Bug
Reporter: Tom Bentley
The command to describe the default topic config
{noformat}
bin/kafka-configs.sh --zookeeper localhost:2181 \
--describe --entity-type topics --entity-name ''
{noformat}
returns without error, but the equivalent command to alter the default topic
config:
{noformat}
bin/kafka-configs.sh --zookeeper localhost:2181 --alter \
--entity-type topics --entity-name '' --add-config retention.ms=1000
{noformat}
returns an error:
{noformat}
Error while executing config command Topic name "" is illegal, it
contains a character other than ASCII alphanumerics, '.', '_' and '-'
org.apache.kafka.common.errors.InvalidTopicException: Topic name "" is
illegal, it contains a character other than ASCII alphanumerics, '.', '_' and
'-'
at org.apache.kafka.common.internals.Topic.validate(Topic.java:45)
at kafka.admin.AdminUtils$.validateTopicConfig(AdminUtils.scala:578)
at kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:595)
at kafka.admin.AdminUtilities$class.changeConfigs(AdminUtils.scala:52)
at kafka.admin.AdminUtils$.changeConfigs(AdminUtils.scala:63)
at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:103)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:70)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6050) Cannot alter default topic config
[
https://issues.apache.org/jira/browse/KAFKA-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200017#comment-16200017
]
huxihx commented on KAFKA-6050:
---
Do you mean your topic name is ""? You are not allowed to create
topics with such names.
> Cannot alter default topic config
> -
>
> Key: KAFKA-6050
> URL: https://issues.apache.org/jira/browse/KAFKA-6050
> Project: Kafka
> Issue Type: Bug
>Reporter: Tom Bentley
>
> The command to describe the default topic config
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 \
> --describe --entity-type topics --entity-name ''
> {noformat}
> returns without error, but the equivalent command to alter the default topic
> config:
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter \
> --entity-type topics --entity-name '' --add-config
> retention.ms=1000
> {noformat}
> returns an error:
> {noformat}
> Error while executing config command Topic name "" is illegal, it
> contains a character other than ASCII alphanumerics, '.', '_' and '-'
> org.apache.kafka.common.errors.InvalidTopicException: Topic name ""
> is illegal, it contains a character other than ASCII alphanumerics, '.', '_'
> and '-'
> at org.apache.kafka.common.internals.Topic.validate(Topic.java:45)
> at kafka.admin.AdminUtils$.validateTopicConfig(AdminUtils.scala:578)
> at kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:595)
> at kafka.admin.AdminUtilities$class.changeConfigs(AdminUtils.scala:52)
> at kafka.admin.AdminUtils$.changeConfigs(AdminUtils.scala:63)
> at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:103)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:70)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6050) Cannot alter default topic config
[
https://issues.apache.org/jira/browse/KAFKA-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200024#comment-16200024
]
Tom Bentley commented on KAFKA-6050:
I'm not trying to create a topic with that name, I'm trying to change the
default for a topic config. "" is used as a "placeholder" topic to
store the default topic configurations (which explains why I'm able to
{{--describe}} it).
> Cannot alter default topic config
> -
>
> Key: KAFKA-6050
> URL: https://issues.apache.org/jira/browse/KAFKA-6050
> Project: Kafka
> Issue Type: Bug
>Reporter: Tom Bentley
>
> The command to describe the default topic config
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 \
> --describe --entity-type topics --entity-name ''
> {noformat}
> returns without error, but the equivalent command to alter the default topic
> config:
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter \
> --entity-type topics --entity-name '' --add-config
> retention.ms=1000
> {noformat}
> returns an error:
> {noformat}
> Error while executing config command Topic name "" is illegal, it
> contains a character other than ASCII alphanumerics, '.', '_' and '-'
> org.apache.kafka.common.errors.InvalidTopicException: Topic name ""
> is illegal, it contains a character other than ASCII alphanumerics, '.', '_'
> and '-'
> at org.apache.kafka.common.internals.Topic.validate(Topic.java:45)
> at kafka.admin.AdminUtils$.validateTopicConfig(AdminUtils.scala:578)
> at kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:595)
> at kafka.admin.AdminUtilities$class.changeConfigs(AdminUtils.scala:52)
> at kafka.admin.AdminUtils$.changeConfigs(AdminUtils.scala:63)
> at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:103)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:70)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6050) Cannot alter default topic config
[
https://issues.apache.org/jira/browse/KAFKA-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200030#comment-16200030
]
huxihx commented on KAFKA-6050:
---
Sorry for the misunderstanding although the exceptions do complain the invalid
topic names:)
> Cannot alter default topic config
> -
>
> Key: KAFKA-6050
> URL: https://issues.apache.org/jira/browse/KAFKA-6050
> Project: Kafka
> Issue Type: Bug
>Reporter: Tom Bentley
>
> The command to describe the default topic config
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 \
> --describe --entity-type topics --entity-name ''
> {noformat}
> returns without error, but the equivalent command to alter the default topic
> config:
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter \
> --entity-type topics --entity-name '' --add-config
> retention.ms=1000
> {noformat}
> returns an error:
> {noformat}
> Error while executing config command Topic name "" is illegal, it
> contains a character other than ASCII alphanumerics, '.', '_' and '-'
> org.apache.kafka.common.errors.InvalidTopicException: Topic name ""
> is illegal, it contains a character other than ASCII alphanumerics, '.', '_'
> and '-'
> at org.apache.kafka.common.internals.Topic.validate(Topic.java:45)
> at kafka.admin.AdminUtils$.validateTopicConfig(AdminUtils.scala:578)
> at kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:595)
> at kafka.admin.AdminUtilities$class.changeConfigs(AdminUtils.scala:52)
> at kafka.admin.AdminUtils$.changeConfigs(AdminUtils.scala:63)
> at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:103)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:70)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6050) Cannot alter default topic config
[
https://issues.apache.org/jira/browse/KAFKA-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200035#comment-16200035
]
Tom Bentley commented on KAFKA-6050:
[~huxi_2b] no worries, I didn't know about this magic topic name either, before
discovering it in the code.
> Cannot alter default topic config
> -
>
> Key: KAFKA-6050
> URL: https://issues.apache.org/jira/browse/KAFKA-6050
> Project: Kafka
> Issue Type: Bug
>Reporter: Tom Bentley
>
> The command to describe the default topic config
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 \
> --describe --entity-type topics --entity-name ''
> {noformat}
> returns without error, but the equivalent command to alter the default topic
> config:
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter \
> --entity-type topics --entity-name '' --add-config
> retention.ms=1000
> {noformat}
> returns an error:
> {noformat}
> Error while executing config command Topic name "" is illegal, it
> contains a character other than ASCII alphanumerics, '.', '_' and '-'
> org.apache.kafka.common.errors.InvalidTopicException: Topic name ""
> is illegal, it contains a character other than ASCII alphanumerics, '.', '_'
> and '-'
> at org.apache.kafka.common.internals.Topic.validate(Topic.java:45)
> at kafka.admin.AdminUtils$.validateTopicConfig(AdminUtils.scala:578)
> at kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:595)
> at kafka.admin.AdminUtilities$class.changeConfigs(AdminUtils.scala:52)
> at kafka.admin.AdminUtils$.changeConfigs(AdminUtils.scala:63)
> at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:103)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:70)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-3355) GetOffsetShell command doesn't work with SASL enabled Kafka
[ https://issues.apache.org/jira/browse/KAFKA-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200057#comment-16200057 ] Pablo Panero commented on KAFKA-3355: - Hello I tested with 0.10.2.1 and is not working. Are there any intentions on merging the proposed PR. If not I am willing on working on it. I would just need access to the CI servers to check why is it failing in two of the options. PR: https://github.com/apache/kafka/pull/2891 [~Sagar Lad] [~fredji] [~magarmes] > GetOffsetShell command doesn't work with SASL enabled Kafka > --- > > Key: KAFKA-3355 > URL: https://issues.apache.org/jira/browse/KAFKA-3355 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 0.9.0.1 > Environment: Kafka 0.9.0.1 >Reporter: TAO XIAO >Assignee: Ashish Singh > > I found that GetOffsetShell doesn't work with SASL enabled Kafka. I believe > this is due to old producer being used in GetOffsetShell. > Kafka version 0.9.0.1 > Exception > % bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list > localhost:9092 --topic test --time -1 > [2016-03-04 21:43:56,597] INFO Verifying properties > (kafka.utils.VerifiableProperties) > [2016-03-04 21:43:56,613] INFO Property client.id is overridden to > GetOffsetShell (kafka.utils.VerifiableProperties) > [2016-03-04 21:43:56,613] INFO Property metadata.broker.list is overridden to > localhost:9092 (kafka.utils.VerifiableProperties) > [2016-03-04 21:43:56,613] INFO Property request.timeout.ms is overridden to > 1000 (kafka.utils.VerifiableProperties) > [2016-03-04 21:43:56,674] INFO Fetching metadata from broker > BrokerEndPoint(0,localhost,9092) with correlation id 0 for 1 topic(s) > Set(test) (kafka.client.ClientUtils$) > [2016-03-04 21:43:56,689] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2016-03-04 21:43:56,705] WARN Fetching topic metadata with correlation id 0 > for topics [Set(test)] from broker [BrokerEndPoint(0,localhost,9092)] failed > (kafka.client.ClientUtils$) > java.nio.BufferUnderflowException > at java.nio.Buffer.nextGetIndex(Buffer.java:498) > at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:304) > at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:36) > at kafka.cluster.BrokerEndPoint$.readFrom(BrokerEndPoint.scala:52) > at > kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28) > at > kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > at scala.collection.immutable.Range.foreach(Range.scala:166) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:28) > at kafka.producer.SyncProducer.send(SyncProducer.scala:120) > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94) > at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:78) > at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-3355) GetOffsetShell command doesn't work with SASL enabled Kafka
[ https://issues.apache.org/jira/browse/KAFKA-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200057#comment-16200057 ] Pablo Panero edited comment on KAFKA-3355 at 10/11/17 10:08 AM: Hello I tested with 0.10.2.1 and is not working. Are there any intentions on merging the proposed PR. If not I am willing on working on it. I would just need access to the CI servers to check why is it failing in two of the options. It would be really helpful for me PR: https://github.com/apache/kafka/pull/2891 [~Sagar Lad] [~fredji] [~magarmes] [~ashishujjain] was (Author: ppanero): Hello I tested with 0.10.2.1 and is not working. Are there any intentions on merging the proposed PR. If not I am willing on working on it. I would just need access to the CI servers to check why is it failing in two of the options. PR: https://github.com/apache/kafka/pull/2891 [~Sagar Lad] [~fredji] [~magarmes] > GetOffsetShell command doesn't work with SASL enabled Kafka > --- > > Key: KAFKA-3355 > URL: https://issues.apache.org/jira/browse/KAFKA-3355 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 0.9.0.1 > Environment: Kafka 0.9.0.1 >Reporter: TAO XIAO >Assignee: Ashish Singh > > I found that GetOffsetShell doesn't work with SASL enabled Kafka. I believe > this is due to old producer being used in GetOffsetShell. > Kafka version 0.9.0.1 > Exception > % bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list > localhost:9092 --topic test --time -1 > [2016-03-04 21:43:56,597] INFO Verifying properties > (kafka.utils.VerifiableProperties) > [2016-03-04 21:43:56,613] INFO Property client.id is overridden to > GetOffsetShell (kafka.utils.VerifiableProperties) > [2016-03-04 21:43:56,613] INFO Property metadata.broker.list is overridden to > localhost:9092 (kafka.utils.VerifiableProperties) > [2016-03-04 21:43:56,613] INFO Property request.timeout.ms is overridden to > 1000 (kafka.utils.VerifiableProperties) > [2016-03-04 21:43:56,674] INFO Fetching metadata from broker > BrokerEndPoint(0,localhost,9092) with correlation id 0 for 1 topic(s) > Set(test) (kafka.client.ClientUtils$) > [2016-03-04 21:43:56,689] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2016-03-04 21:43:56,705] WARN Fetching topic metadata with correlation id 0 > for topics [Set(test)] from broker [BrokerEndPoint(0,localhost,9092)] failed > (kafka.client.ClientUtils$) > java.nio.BufferUnderflowException > at java.nio.Buffer.nextGetIndex(Buffer.java:498) > at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:304) > at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:36) > at kafka.cluster.BrokerEndPoint$.readFrom(BrokerEndPoint.scala:52) > at > kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28) > at > kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > at scala.collection.immutable.Range.foreach(Range.scala:166) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:28) > at kafka.producer.SyncProducer.send(SyncProducer.scala:120) > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94) > at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:78) > at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6051) ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on shutdown
Maytee Chinavanichkit created KAFKA-6051:
Summary: ReplicaFetcherThread should close the
ReplicaFetcherBlockingSend earlier on shutdown
Key: KAFKA-6051
URL: https://issues.apache.org/jira/browse/KAFKA-6051
Project: Kafka
Issue Type: Bug
Reporter: Maytee Chinavanichkit
The ReplicaFetcherBlockingSend works as designed and will blocks until it is
able to get data. This becomes a problem when we are gracefully shutting down a
broker. The controller will attempt to shutdown the fetchers and elect new
leaders. When the last fetch of partition is removed, as part of the
{replicaManager.becomeLeaderOrFollower} call will proceed to shut down any idle
ReplicaFetcherThread. The shutdown process here can block up to until the last
fetch request completes. This blocking delay is a big problem because the
{replicaStateChangeLock}, and {mapLock} in {AbstractFetcherManager} is still
locked causing latency spikes on multiple brokers.
At this point in time, we do not need the last response as the fetcher is
shutting down. We should close the leaderEndpoint early during
{initiateShutdown()} instead of after {super.shutdown()}.
For example we see here the shutdown blocked the broker from processing more
replica changes for ~500 ms
{code}
[2017-09-01 18:11:42,879] INFO [ReplicaFetcherThread-0-2], Shutting down
(kafka.server.ReplicaFetcherThread)
[2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Stopped
(kafka.server.ReplicaFetcherThread)
[2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Shutdown completed
(kafka.server.ReplicaFetcherThread)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6051) ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on shutdown
[
https://issues.apache.org/jira/browse/KAFKA-6051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200088#comment-16200088
]
ASF GitHub Bot commented on KAFKA-6051:
---
GitHub user mayt opened a pull request:
https://github.com/apache/kafka/pull/4056
KAFKA-6051 Close the ReplicaFetcherBlockingSend earlier on shutdown
Rearranged the testAddPartitionDuringDeleteTopic() test to keep the
likelyhood of the race condition.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mayt/kafka KAFKA-6051
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/4056.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4056
commit 36c1fa6ca3bab4dc070910cba9223f4141982d82
Author: Maytee Chinavanichkit
Date: 2017-10-11T10:35:54Z
KAFKA-6051 Close the ReplicaFetcherBlockingSend earlier on shutdown
Rearranged the testAddPartitionDuringDeleteTopic() test to keep the
likelyhood of the race condition.
> ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on
> shutdown
>
>
> Key: KAFKA-6051
> URL: https://issues.apache.org/jira/browse/KAFKA-6051
> Project: Kafka
> Issue Type: Bug
>Reporter: Maytee Chinavanichkit
>
> The ReplicaFetcherBlockingSend works as designed and will blocks until it is
> able to get data. This becomes a problem when we are gracefully shutting down
> a broker. The controller will attempt to shutdown the fetchers and elect new
> leaders. When the last fetch of partition is removed, as part of the
> {replicaManager.becomeLeaderOrFollower} call will proceed to shut down any
> idle ReplicaFetcherThread. The shutdown process here can block up to until
> the last fetch request completes. This blocking delay is a big problem
> because the {replicaStateChangeLock}, and {mapLock} in
> {AbstractFetcherManager} is still locked causing latency spikes on multiple
> brokers.
> At this point in time, we do not need the last response as the fetcher is
> shutting down. We should close the leaderEndpoint early during
> {initiateShutdown()} instead of after {super.shutdown()}.
> For example we see here the shutdown blocked the broker from processing more
> replica changes for ~500 ms
> {code}
> [2017-09-01 18:11:42,879] INFO [ReplicaFetcherThread-0-2], Shutting down
> (kafka.server.ReplicaFetcherThread)
> [2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Stopped
> (kafka.server.ReplicaFetcherThread)
> [2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Shutdown completed
> (kafka.server.ReplicaFetcherThread)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (KAFKA-6051) ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on shutdown
[
https://issues.apache.org/jira/browse/KAFKA-6051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maytee Chinavanichkit updated KAFKA-6051:
-
Comment: was deleted
(was: https://github.com/apache/kafka/pull/4056)
> ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on
> shutdown
>
>
> Key: KAFKA-6051
> URL: https://issues.apache.org/jira/browse/KAFKA-6051
> Project: Kafka
> Issue Type: Bug
>Reporter: Maytee Chinavanichkit
>
> The ReplicaFetcherBlockingSend works as designed and will blocks until it is
> able to get data. This becomes a problem when we are gracefully shutting down
> a broker. The controller will attempt to shutdown the fetchers and elect new
> leaders. When the last fetch of partition is removed, as part of the
> {replicaManager.becomeLeaderOrFollower} call will proceed to shut down any
> idle ReplicaFetcherThread. The shutdown process here can block up to until
> the last fetch request completes. This blocking delay is a big problem
> because the {replicaStateChangeLock}, and {mapLock} in
> {AbstractFetcherManager} is still locked causing latency spikes on multiple
> brokers.
> At this point in time, we do not need the last response as the fetcher is
> shutting down. We should close the leaderEndpoint early during
> {initiateShutdown()} instead of after {super.shutdown()}.
> For example we see here the shutdown blocked the broker from processing more
> replica changes for ~500 ms
> {code}
> [2017-09-01 18:11:42,879] INFO [ReplicaFetcherThread-0-2], Shutting down
> (kafka.server.ReplicaFetcherThread)
> [2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Stopped
> (kafka.server.ReplicaFetcherThread)
> [2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Shutdown completed
> (kafka.server.ReplicaFetcherThread)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6051) ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on shutdown
[
https://issues.apache.org/jira/browse/KAFKA-6051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maytee Chinavanichkit updated KAFKA-6051:
-
Description:
The ReplicaFetcherBlockingSend works as designed and will blocks until it is
able to get data. This becomes a problem when we are gracefully shutting down a
broker. The controller will attempt to shutdown the fetchers and elect new
leaders. When the last fetch of partition is removed, as part of the
{{replicaManager.becomeLeaderOrFollower}} call will proceed to shut down any
idle ReplicaFetcherThread. The shutdown process here can block up to until the
last fetch request completes. This blocking delay is a big problem because the
{{replicaStateChangeLock}}, and {{mapLock}} in {{AbstractFetcherManager}} is
still locked causing latency spikes on multiple brokers.
At this point in time, we do not need the last response as the fetcher is
shutting down. We should close the leaderEndpoint early during
{{initiateShutdown()}} instead of after {{super.shutdown()}}.
For example we see here the shutdown blocked the broker from processing more
replica changes for ~500 ms
{code}
[2017-09-01 18:11:42,879] INFO [ReplicaFetcherThread-0-2], Shutting down
(kafka.server.ReplicaFetcherThread)
[2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Stopped
(kafka.server.ReplicaFetcherThread)
[2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Shutdown completed
(kafka.server.ReplicaFetcherThread)
{code}
was:
The ReplicaFetcherBlockingSend works as designed and will blocks until it is
able to get data. This becomes a problem when we are gracefully shutting down a
broker. The controller will attempt to shutdown the fetchers and elect new
leaders. When the last fetch of partition is removed, as part of the
{replicaManager.becomeLeaderOrFollower} call will proceed to shut down any idle
ReplicaFetcherThread. The shutdown process here can block up to until the last
fetch request completes. This blocking delay is a big problem because the
{replicaStateChangeLock}, and {mapLock} in {AbstractFetcherManager} is still
locked causing latency spikes on multiple brokers.
At this point in time, we do not need the last response as the fetcher is
shutting down. We should close the leaderEndpoint early during
{initiateShutdown()} instead of after {super.shutdown()}.
For example we see here the shutdown blocked the broker from processing more
replica changes for ~500 ms
{code}
[2017-09-01 18:11:42,879] INFO [ReplicaFetcherThread-0-2], Shutting down
(kafka.server.ReplicaFetcherThread)
[2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Stopped
(kafka.server.ReplicaFetcherThread)
[2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Shutdown completed
(kafka.server.ReplicaFetcherThread)
{code}
> ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on
> shutdown
>
>
> Key: KAFKA-6051
> URL: https://issues.apache.org/jira/browse/KAFKA-6051
> Project: Kafka
> Issue Type: Bug
>Reporter: Maytee Chinavanichkit
>
> The ReplicaFetcherBlockingSend works as designed and will blocks until it is
> able to get data. This becomes a problem when we are gracefully shutting down
> a broker. The controller will attempt to shutdown the fetchers and elect new
> leaders. When the last fetch of partition is removed, as part of the
> {{replicaManager.becomeLeaderOrFollower}} call will proceed to shut down any
> idle ReplicaFetcherThread. The shutdown process here can block up to until
> the last fetch request completes. This blocking delay is a big problem
> because the {{replicaStateChangeLock}}, and {{mapLock}} in
> {{AbstractFetcherManager}} is still locked causing latency spikes on multiple
> brokers.
> At this point in time, we do not need the last response as the fetcher is
> shutting down. We should close the leaderEndpoint early during
> {{initiateShutdown()}} instead of after {{super.shutdown()}}.
> For example we see here the shutdown blocked the broker from processing more
> replica changes for ~500 ms
> {code}
> [2017-09-01 18:11:42,879] INFO [ReplicaFetcherThread-0-2], Shutting down
> (kafka.server.ReplicaFetcherThread)
> [2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Stopped
> (kafka.server.ReplicaFetcherThread)
> [2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Shutdown completed
> (kafka.server.ReplicaFetcherThread)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6051) ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on shutdown
[
https://issues.apache.org/jira/browse/KAFKA-6051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200091#comment-16200091
]
Maytee Chinavanichkit commented on KAFKA-6051:
--
https://github.com/apache/kafka/pull/4056
> ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on
> shutdown
>
>
> Key: KAFKA-6051
> URL: https://issues.apache.org/jira/browse/KAFKA-6051
> Project: Kafka
> Issue Type: Bug
>Reporter: Maytee Chinavanichkit
>
> The ReplicaFetcherBlockingSend works as designed and will blocks until it is
> able to get data. This becomes a problem when we are gracefully shutting down
> a broker. The controller will attempt to shutdown the fetchers and elect new
> leaders. When the last fetch of partition is removed, as part of the
> {replicaManager.becomeLeaderOrFollower} call will proceed to shut down any
> idle ReplicaFetcherThread. The shutdown process here can block up to until
> the last fetch request completes. This blocking delay is a big problem
> because the {replicaStateChangeLock}, and {mapLock} in
> {AbstractFetcherManager} is still locked causing latency spikes on multiple
> brokers.
> At this point in time, we do not need the last response as the fetcher is
> shutting down. We should close the leaderEndpoint early during
> {initiateShutdown()} instead of after {super.shutdown()}.
> For example we see here the shutdown blocked the broker from processing more
> replica changes for ~500 ms
> {code}
> [2017-09-01 18:11:42,879] INFO [ReplicaFetcherThread-0-2], Shutting down
> (kafka.server.ReplicaFetcherThread)
> [2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Stopped
> (kafka.server.ReplicaFetcherThread)
> [2017-09-01 18:11:43,314] INFO [ReplicaFetcherThread-0-2], Shutdown completed
> (kafka.server.ReplicaFetcherThread)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-6052) Consumers not polling when isolation.level=read_committed
Ansel Zandegran created KAFKA-6052:
--
Summary: Consumers not polling when isolation.level=read_committed
Key: KAFKA-6052
URL: https://issues.apache.org/jira/browse/KAFKA-6052
Project: Kafka
Issue Type: Bug
Components: consumer, producer
Affects Versions: 0.11.0.0
Environment: Windows 10. All processes running in embedded mode
Reporter: Ansel Zandegran
Attachments: logFile.log
I am trying to send a transactional record with exactly once schematics. These
are my producer, consumer and broker setups.
public void sendWithTTemp(String topic, EHEvent event) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"localhost:9092,localhost:9093,localhost:9094");
//props.put("bootstrap.servers",
"34.240.248.190:9092,52.50.95.30:9092,52.50.95.30:9092");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "1");
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);
props.put(ProducerConfig.LINGER_MS_CONFIG, 1);
props.put("transactional.id", "TID" + transactionId.incrementAndGet());
props.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, "5000");
Producer producer =
new KafkaProducer<>(props,
new StringSerializer(),
new StringSerializer());
Logger.log(this, "Initializing transaction...");
producer.initTransactions();
Logger.log(this, "Initializing done.");
try {
Logger.log(this, "Begin transaction...");
producer.beginTransaction();
Logger.log(this, "Begin transaction done.");
Logger.log(this, "Sending events...");
producer.send(new ProducerRecord<>(topic,
event.getKey().toString(),
event.getValue().toString()));
Logger.log(this, "Sending events done.");
Logger.log(this, "Committing...");
producer.commitTransaction();
Logger.log(this, "Committing done.");
} catch (ProducerFencedException | OutOfOrderSequenceException
| AuthorizationException e) {
producer.close();
e.printStackTrace();
} catch (KafkaException e) {
producer.abortTransaction();
e.printStackTrace();
}
producer.close();
}
*In Consumer*
I have set isolation.level=read_committed
*In 3 Brokers*
I'm running with the following properties
Properties props = new Properties();
props.setProperty("broker.id", "" + i);
props.setProperty("listeners", "PLAINTEXT://:909" + (2 + i));
props.setProperty("log.dirs", Configuration.KAFKA_DATA_PATH + "\\B" + i);
props.setProperty("num.partitions", "1");
props.setProperty("zookeeper.connect", "localhost:2181");
props.setProperty("zookeeper.connection.timeout.ms", "6000");
props.setProperty("min.insync.replicas", "2");
props.setProperty("offsets.topic.replication.factor", "2");
props.setProperty("offsets.topic.num.partitions", "1");
props.setProperty("transaction.state.log.num.partitions", "2");
props.setProperty("transaction.state.log.replication.factor", "2");
props.setProperty("transaction.state.log.min.isr", "2");
I am not getting any records in the consumer. When I set
isolation.level=read_uncommitted, I get the records. I assume that the records
are not getting commited. What could be the problem? log attached
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6028) Improve the quota throttle communication.
[ https://issues.apache.org/jira/browse/KAFKA-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200475#comment-16200475 ] Jun Rao commented on KAFKA-6028: With a reduced window, it does mean that there is a lower bound of quota that one can set. For example, given the max fetch response size of 10MB and a 10 sec window, it means that a user can't set a quota smaller than 1MB/sec. > Improve the quota throttle communication. > - > > Key: KAFKA-6028 > URL: https://issues.apache.org/jira/browse/KAFKA-6028 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Affects Versions: 1.0.0 >Reporter: Jiangjie Qin >Assignee: Jiangjie Qin > Fix For: 1.1.0 > > > Currently if a client is throttled duet to quota violation, the broker will > only send back a response to the clients after the throttle time has passed. > In this case, the clients don't know how long the response will be throttled > and might hit request timeout before the response is returned. As a result > the clients will retry sending a request and results a even longer throttle > time. > The above scenario could happen when a large clients group sending records to > the brokers. We saw this when a MapReduce job pushes data to the Kafka > cluster. > To improve this, the broker can return the response with throttle time > immediately after processing the requests. After that, the broker will mute > the channel for this client. A correct client implementation should back off > for that long before sending the next request. If the client ignored the > throttle time and send the next request immediately, the channel will be > muted and the request won't be processed until the throttle time has passed. > A KIP will follow with more details. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6007) Connect can't validate against transforms in plugins.path
[
https://issues.apache.org/jira/browse/KAFKA-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200544#comment-16200544
]
Konstantine Karantasis commented on KAFKA-6007:
---
The most promising approach to generally address this issue of mixing and
matching plugins from within the {{plugin.path}} and the {{CLASSPATH}} seems to
be writing a more specialized classloader that would wrap the system
classloader (currently used as is as another classloader).
> Connect can't validate against transforms in plugins.path
> -
>
> Key: KAFKA-6007
> URL: https://issues.apache.org/jira/browse/KAFKA-6007
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.11.0.0, 0.11.0.1, 1.0.0
>Reporter: Stephane Maarek
>Assignee: Konstantine Karantasis
> Fix For: 0.11.0.2, 1.0.1
>
>
> Kafka Connect can't validate a custom transformation if placed in plugins
> path.
> Here's the output I get on the validate call:
> {code:java}
> Invalid value com.mycorp.kafka.transforms.impl.FlattenSinkRecord for
> configuration transforms.Flat.type: Class
> com.mycorp.kafka.transforms.impl.FlattenSinkRecord could not be found.
> Invalid value null for configuration transforms.Flat.type: Not a
> Transformation
> "recommended_values": [
> "com.mycorp.kafka.transforms.Flatten$Key",
> "com.mycorp.kafka.transforms.Flatten$Value",
> "com.mycorp.kafka.transforms.impl.FlattenSinkRecord",
> "org.apache.kafka.connect.transforms.Cast$Key",
> "org.apache.kafka.connect.transforms.Cast$Value",
> "org.apache.kafka.connect.transforms.ExtractField$Key",
> "org.apache.kafka.connect.transforms.ExtractField$Value",
> "org.apache.kafka.connect.transforms.Flatten$Key",
> "org.apache.kafka.connect.transforms.Flatten$Value",
> "org.apache.kafka.connect.transforms.HoistField$Key",
> "org.apache.kafka.connect.transforms.HoistField$Value",
> "org.apache.kafka.connect.transforms.InsertField$Key",
> "org.apache.kafka.connect.transforms.InsertField$Value",
> "org.apache.kafka.connect.transforms.MaskField$Key",
> "org.apache.kafka.connect.transforms.MaskField$Value",
> "org.apache.kafka.connect.transforms.RegexRouter",
> "org.apache.kafka.connect.transforms.ReplaceField$Key",
> "org.apache.kafka.connect.transforms.ReplaceField$Value",
> "org.apache.kafka.connect.transforms.SetSchemaMetadata$Key",
> "org.apache.kafka.connect.transforms.SetSchemaMetadata$Value",
> "org.apache.kafka.connect.transforms.TimestampConverter$Key",
> "org.apache.kafka.connect.transforms.TimestampConverter$Value",
> "org.apache.kafka.connect.transforms.TimestampRouter",
> "org.apache.kafka.connect.transforms.ValueToKey"],
> {code}
> As you can see the class appear in the recommended values (!) but can't be
> picked up on the validate call.
> I believe it's because the recommender implements class discovery using
> plugins:
> https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectorConfig.java#L194
> But the class inference itself doesn't:
> https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectorConfig.java#L199
> (I'm not an expert in class loading though, just a guess... Unsure how to fix)
> A quick fix is to add the transformations in the ClassPath itself, but that
> defeats the point a bit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-6053) NoSuchMethodError when creating ProducerRecord in upgrade system tests
Apurva Mehta created KAFKA-6053:
---
Summary: NoSuchMethodError when creating ProducerRecord in upgrade
system tests
Key: KAFKA-6053
URL: https://issues.apache.org/jira/browse/KAFKA-6053
Project: Kafka
Issue Type: Bug
Reporter: Apurva Mehta
Assignee: Apurva Mehta
This patch https://github.com/apache/kafka/pull/4029 used a new constructor for
{{ProducerRecord}} which doesn't exist in older clients. Hence system tests
which use older clients fail with:
{noformat}
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.kafka.clients.producer.ProducerRecord.(Ljava/lang/String;Ljava/lang/Integer;Ljava/lang/Long;Ljava/lang/Object;Ljava/lang/Object;)V
at
org.apache.kafka.tools.VerifiableProducer.send(VerifiableProducer.java:232)
at
org.apache.kafka.tools.VerifiableProducer.run(VerifiableProducer.java:462)
at
org.apache.kafka.tools.VerifiableProducer.main(VerifiableProducer.java:500)
{"timestamp":1507711495458,"name":"shutdown_complete"}
{"timestamp":1507711495459,"name":"tool_data","sent":0,"acked":0,"target_throughput":1,"avg_throughput":0.0}
amehta-macbook-pro:worker6 apurva$
{noformat}
A trivial fix is to only use the new constructor if a message create time is
explicitly passed to the VerifiableProducer, since older versions of the test
would never use it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-6054) ERROR "SubscriptionInfo - unable to decode subscription data: version=2" when upgrading from 0.10.0.0 to 0.10.2.1
James Cheng created KAFKA-6054:
--
Summary: ERROR "SubscriptionInfo - unable to decode subscription
data: version=2" when upgrading from 0.10.0.0 to 0.10.2.1
Key: KAFKA-6054
URL: https://issues.apache.org/jira/browse/KAFKA-6054
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 0.10.2.1
Reporter: James Cheng
We upgraded an app from kafka-streams 0.10.0.0 to 0.10.2.1. We did a rolling
upgrade of the app, so that one point, there were both 0.10.0.0-based instances
and 0.10.2.1-based instances running.
We observed the following stack trace:
{code}
2017-10-11 07:02:19.964 [StreamThread-3] ERROR o.a.k.s.p.i.a.SubscriptionInfo -
unable to decode subscription data: version=2
org.apache.kafka.streams.errors.TaskAssignmentException: unable to decode
subscription data: version=2
at
org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfo.decode(SubscriptionInfo.java:113)
at
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.assign(StreamPartitionAssignor.java:235)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:260)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:404)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$900(AbstractCoordinator.java:81)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:358)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:340)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:679)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658)
at
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:243)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:345)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:977)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:937)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:295)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:218)
{code}
I spoke with [~mjsax] and he said this is a known issue that happens when you
have both 0.10.0.0 instances and 0.10.2.1 instances running at the same time,
because the internal version number of the protocol changed when adding
Interactive Queries. Matthias asked me to file this JIRA>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6054) ERROR "SubscriptionInfo - unable to decode subscription data: version=2" when upgrading from 0.10.0.0 to 0.10.2.1
[
https://issues.apache.org/jira/browse/KAFKA-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200571#comment-16200571
]
James Cheng commented on KAFKA-6054:
Here is my conversation with [~mjsax] from the Confluent Slack channel:
{quote}
James Cheng [9:16 AM]
Does this stack trace mean anything to anyone? It happened when we upgraded a
kafka streams app from 0.10.0.0 to 0.10.2.1.
^ @mjsax, if you have any time to look. Thanks.
Matthias J Sax
[9:20 AM]
That makes sense. We bumped the internal version number when adding IQ feature
-- thus, it seems you cannot mix instances for both version.
[9:21]
Seems, we messed up the upgrade path :disappointed:
[9:21]
If you can, you would need to stop all old instances, before starting with the
new version.
[9:21]
Can you also open a JIRA for this?
[9:24]
Thus, rolling bounces to upgrade should actually work -- is this what you are
doing?
James Cheng [9:27 AM]
Yes, we're doing a rolling upgrade. We had (at one point, at least) both
instances running.
[9:27]
I imagine that if the 0.10.0.0 versions crashed, then restarted running
0.10.2.1, then they would be fine because they are all the same version at that
point, right?
Matthias J Sax
[9:27 AM]
Yes.
James Cheng [9:27 AM]
Cool, thanks.
Matthias J Sax
[9:28 AM]
Anyway. Please file a JIRA -- upgrading should always work without this error.
James Cheng [9:29 AM]
I'll file the JIRA.
Matthias J Sax
[9:30 AM]
Thx.
{quote}
> ERROR "SubscriptionInfo - unable to decode subscription data: version=2" when
> upgrading from 0.10.0.0 to 0.10.2.1
> -
>
> Key: KAFKA-6054
> URL: https://issues.apache.org/jira/browse/KAFKA-6054
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.2.1
>Reporter: James Cheng
>
> We upgraded an app from kafka-streams 0.10.0.0 to 0.10.2.1. We did a rolling
> upgrade of the app, so that one point, there were both 0.10.0.0-based
> instances and 0.10.2.1-based instances running.
> We observed the following stack trace:
> {code}
> 2017-10-11 07:02:19.964 [StreamThread-3] ERROR o.a.k.s.p.i.a.SubscriptionInfo
> -
> unable to decode subscription data: version=2
> org.apache.kafka.streams.errors.TaskAssignmentException: unable to decode
> subscription data: version=2
> at
> org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfo.decode(SubscriptionInfo.java:113)
> at
> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.assign(StreamPartitionAssignor.java:235)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:260)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:404)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$900(AbstractCoordinator.java:81)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:358)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:340)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:679)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:243)
> at
> org.apache.kafka.clients.consumer.internals.Consum
[jira] [Commented] (KAFKA-5877) Controller should only update reassignment znode if there is change in the reassignment data
[ https://issues.apache.org/jira/browse/KAFKA-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200574#comment-16200574 ] Fredrik Vraalsen commented on KAFKA-5877: - Is the fix also being considered for any earlier bugfixes, e.g. 0.11.0.2? > Controller should only update reassignment znode if there is change in the > reassignment data > > > Key: KAFKA-5877 > URL: https://issues.apache.org/jira/browse/KAFKA-5877 > Project: Kafka > Issue Type: Bug >Reporter: Dong Lin >Assignee: Dong Lin > Fix For: 1.1.0 > > > I encountered a scenario where controller keeps printing the following stack > trace repeatedly for a finite set of partitions. Although I have not fully > figured out the cause of this event, it seems that controller will update the > reassignment znode even if the new data is same as existing data. This patch > optimizes the controller behavior by only updating reassignment znode if it > needs to change the reassignment znode data. > 2017/09/12 20:34:05.842 [KafkaController] [Controller 1376005]: Error > completing reassignment of partition [FederatorResultEvent,202] > kafka.common.KafkaException: Partition [FederatorResultEvent,202] to be > reassigned is already assigned to replicas 1367001,1384010,1386010. Ignoring > request for partition reassignment > at > kafka.controller.KafkaController.initiateReassignReplicasForTopicPartition(KafkaController.scala:608) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.KafkaController$PartitionReassignment$$anonfun$process$14.apply(KafkaController.scala:1327) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.KafkaController$PartitionReassignment$$anonfun$process$14.apply(KafkaController.scala:1320) > ~[kafka_2.10-0.11.0.9.jar:?] > at > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224) > ~[scala-library-2.10.4.jar:?] > at > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > ~[scala-library-2.10.4.jar:?] > at > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > ~[scala-library-2.10.4.jar:?] > at > kafka.controller.KafkaController$PartitionReassignment.process(KafkaController.scala:1320) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:53) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:53) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:53) > ~[kafka_2.10-0.11.0.9.jar:?] > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:52) > ~[kafka_2.10-0.11.0.9.jar:?] > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64) > ~[kafka_2.10-0.11.0.9.jar:?] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6053) NoSuchMethodError when creating ProducerRecord in upgrade system tests
[
https://issues.apache.org/jira/browse/KAFKA-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200593#comment-16200593
]
ASF GitHub Bot commented on KAFKA-6053:
---
GitHub user apurvam opened a pull request:
https://github.com/apache/kafka/pull/4057
KAFKA-6053: Fix NoSuchMethodError when creating ProducerRecords with older
client versions
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apurvam/kafka
KAFKA-6053-fix-no-such-method-error-in-producer-record
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/4057.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4057
commit 221a96da1dc6221dd3f61786d5fc1119b6848a7f
Author: Apurva Mehta
Date: 2017-10-11T16:58:44Z
Fix NoSuchMethodError when creating ProducerRecords with older client
versions
> NoSuchMethodError when creating ProducerRecord in upgrade system tests
> --
>
> Key: KAFKA-6053
> URL: https://issues.apache.org/jira/browse/KAFKA-6053
> Project: Kafka
> Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>
> This patch https://github.com/apache/kafka/pull/4029 used a new constructor
> for {{ProducerRecord}} which doesn't exist in older clients. Hence system
> tests which use older clients fail with:
> {noformat}
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.kafka.clients.producer.ProducerRecord.(Ljava/lang/String;Ljava/lang/Integer;Ljava/lang/Long;Ljava/lang/Object;Ljava/lang/Object;)V
> at
> org.apache.kafka.tools.VerifiableProducer.send(VerifiableProducer.java:232)
> at
> org.apache.kafka.tools.VerifiableProducer.run(VerifiableProducer.java:462)
> at
> org.apache.kafka.tools.VerifiableProducer.main(VerifiableProducer.java:500)
> {"timestamp":1507711495458,"name":"shutdown_complete"}
> {"timestamp":1507711495459,"name":"tool_data","sent":0,"acked":0,"target_throughput":1,"avg_throughput":0.0}
> amehta-macbook-pro:worker6 apurva$
> {noformat}
> A trivial fix is to only use the new constructor if a message create time is
> explicitly passed to the VerifiableProducer, since older versions of the test
> would never use it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5877) Controller should only update reassignment znode if there is change in the reassignment data
[ https://issues.apache.org/jira/browse/KAFKA-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200603#comment-16200603 ] Dong Lin commented on KAFKA-5877: - I am not sure about our policy of back-port. Maybe [~ijuma] has better idea about it. > Controller should only update reassignment znode if there is change in the > reassignment data > > > Key: KAFKA-5877 > URL: https://issues.apache.org/jira/browse/KAFKA-5877 > Project: Kafka > Issue Type: Bug >Reporter: Dong Lin >Assignee: Dong Lin > Fix For: 1.1.0 > > > I encountered a scenario where controller keeps printing the following stack > trace repeatedly for a finite set of partitions. Although I have not fully > figured out the cause of this event, it seems that controller will update the > reassignment znode even if the new data is same as existing data. This patch > optimizes the controller behavior by only updating reassignment znode if it > needs to change the reassignment znode data. > 2017/09/12 20:34:05.842 [KafkaController] [Controller 1376005]: Error > completing reassignment of partition [FederatorResultEvent,202] > kafka.common.KafkaException: Partition [FederatorResultEvent,202] to be > reassigned is already assigned to replicas 1367001,1384010,1386010. Ignoring > request for partition reassignment > at > kafka.controller.KafkaController.initiateReassignReplicasForTopicPartition(KafkaController.scala:608) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.KafkaController$PartitionReassignment$$anonfun$process$14.apply(KafkaController.scala:1327) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.KafkaController$PartitionReassignment$$anonfun$process$14.apply(KafkaController.scala:1320) > ~[kafka_2.10-0.11.0.9.jar:?] > at > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224) > ~[scala-library-2.10.4.jar:?] > at > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > ~[scala-library-2.10.4.jar:?] > at > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > ~[scala-library-2.10.4.jar:?] > at > kafka.controller.KafkaController$PartitionReassignment.process(KafkaController.scala:1320) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:53) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:53) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:53) > ~[kafka_2.10-0.11.0.9.jar:?] > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:52) > ~[kafka_2.10-0.11.0.9.jar:?] > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64) > ~[kafka_2.10-0.11.0.9.jar:?] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6048) Support negative record timestamps
[
https://issues.apache.org/jira/browse/KAFKA-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200726#comment-16200726
]
Matthias J. Sax commented on KAFKA-6048:
You can always start a KIP -- but it will be a large undertaking and long KIP
discussion. Just saying. :)
> Support negative record timestamps
> --
>
> Key: KAFKA-6048
> URL: https://issues.apache.org/jira/browse/KAFKA-6048
> Project: Kafka
> Issue Type: Improvement
> Components: clients, core, streams
>Affects Versions: 1.0.0
>Reporter: Matthias J. Sax
>Assignee: james chien
> Labels: needs-kip
>
> Kafka does not support negative record timestamps, and this prevents the
> storage of historical data in Kafka. In general, negative timestamps are
> supported by UNIX system time stamps:
> From https://en.wikipedia.org/wiki/Unix_time
> {quote}
> The Unix time number is zero at the Unix epoch, and increases by exactly
> 86,400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after
> the epoch, is represented by the Unix time number 12,677 × 86,400 =
> 1095292800. This can be extended backwards from the epoch too, using negative
> numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch, is
> represented by the Unix time number −4,472 × 86,400 = −386380800.
> {quote}
> Allowing for negative timestamps would require multiple changes:
> - while brokers in general do support negative timestamps, broker use {{-1}}
> as default value if a producer uses an old message format (this would not be
> compatible with supporting negative timestamps "end-to-end" as {{-1}} cannot
> be used as "unknown" anymore): we could introduce a message flag indicating a
> missing timestamp (and let producer throw an exception if
> {{ConsumerRecord#timestamp()}} is called. Another possible solution might be,
> to require topics that are used by old producers to be configured with
> {{LogAppendTime}} semantics and rejecting writes to topics with
> {{CreateTime}} semantics for older message formats
> - {{KafkaProducer}} does not allow to send records with negative timestamp
> and thus this would need to be fixed
> - Streams API does drop records with negative timestamps (or fails by
> default) -- also, some internal store implementation for windowed stores
> assume that there are not negative timestamps to do range queries
> There might be other gaps we need to address. This is just a summary of issue
> coming to my mind atm.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6050) Cannot alter default topic config
[
https://issues.apache.org/jira/browse/KAFKA-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200735#comment-16200735
]
Mickael Maison commented on KAFKA-6050:
---
I've not played with {{topics}} entities yet but for {{users}} and {{clients}}
you need to use {{--entity-default}} to specify the {{}} entity.
> Cannot alter default topic config
> -
>
> Key: KAFKA-6050
> URL: https://issues.apache.org/jira/browse/KAFKA-6050
> Project: Kafka
> Issue Type: Bug
>Reporter: Tom Bentley
>
> The command to describe the default topic config
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 \
> --describe --entity-type topics --entity-name ''
> {noformat}
> returns without error, but the equivalent command to alter the default topic
> config:
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter \
> --entity-type topics --entity-name '' --add-config
> retention.ms=1000
> {noformat}
> returns an error:
> {noformat}
> Error while executing config command Topic name "" is illegal, it
> contains a character other than ASCII alphanumerics, '.', '_' and '-'
> org.apache.kafka.common.errors.InvalidTopicException: Topic name ""
> is illegal, it contains a character other than ASCII alphanumerics, '.', '_'
> and '-'
> at org.apache.kafka.common.internals.Topic.validate(Topic.java:45)
> at kafka.admin.AdminUtils$.validateTopicConfig(AdminUtils.scala:578)
> at kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:595)
> at kafka.admin.AdminUtilities$class.changeConfigs(AdminUtils.scala:52)
> at kafka.admin.AdminUtils$.changeConfigs(AdminUtils.scala:63)
> at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:103)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:70)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6022) mirror maker stores offset in zookeeper
[ https://issues.apache.org/jira/browse/KAFKA-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200790#comment-16200790 ] Ronald van de Kuil commented on KAFKA-6022: --- Hi Manikumar, I am glad to have been of help. I did clean the DC2 "cluster" and restarted the mirror maker with the following settings. bootstrap.servers=pi1:9092,pi2:9092 group.id=kafkaMirrorV2 key.deserializer=org.apache.kafka.common.serialization.StringDeserializer value.deserializer=org.apache.kafka.common.serialization.StringDeserializer I see the new kafka based consumer using Kafka Tool but I do not see the offset. The same goes for the command line: pi@pi4:/opt/kafka_2.11-0.11.0.0 $ bin/kafka-consumer-groups.sh --describe --bootstrap-server pi1:9092,pi2:9092 --new-consumer --group kafkaMirrorV2 Nothing shows in the table. The mirror is not being established. It is not a big deal for me because it is just a setup for learning for me. I edited the consumer.properties of the mirror maker a bit in the style of the consumer api. Am I missing something here? > mirror maker stores offset in zookeeper > --- > > Key: KAFKA-6022 > URL: https://issues.apache.org/jira/browse/KAFKA-6022 > Project: Kafka > Issue Type: Improvement >Reporter: Ronald van de Kuil >Priority: Minor > > I happened to notice that the mirror maker stores its offset in zookeeper. > I do not see it using: > bin/kafka-consumer-groups.sh --bootstrap-server pi1:9092 --new-consumer --list > I do however see consumers that store their offset in kafka. > I would guess that storing the offset in zookeeper is old style? > Would it be an idea to update the mirror maker to the new consumer style? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6050) Cannot alter default topic config
[
https://issues.apache.org/jira/browse/KAFKA-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200812#comment-16200812
]
Tom Bentley commented on KAFKA-6050:
Ah, thanks [~mimaison], you're right! In that case I would suggest that trying
to use {{}} as the {{--entity-name}} should consistently print a
helpful error message about using {{--entity-default}}, rather than being
accepted in one situation, and erroring in the other.
> Cannot alter default topic config
> -
>
> Key: KAFKA-6050
> URL: https://issues.apache.org/jira/browse/KAFKA-6050
> Project: Kafka
> Issue Type: Bug
>Reporter: Tom Bentley
>
> The command to describe the default topic config
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 \
> --describe --entity-type topics --entity-name ''
> {noformat}
> returns without error, but the equivalent command to alter the default topic
> config:
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter \
> --entity-type topics --entity-name '' --add-config
> retention.ms=1000
> {noformat}
> returns an error:
> {noformat}
> Error while executing config command Topic name "" is illegal, it
> contains a character other than ASCII alphanumerics, '.', '_' and '-'
> org.apache.kafka.common.errors.InvalidTopicException: Topic name ""
> is illegal, it contains a character other than ASCII alphanumerics, '.', '_'
> and '-'
> at org.apache.kafka.common.internals.Topic.validate(Topic.java:45)
> at kafka.admin.AdminUtils$.validateTopicConfig(AdminUtils.scala:578)
> at kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:595)
> at kafka.admin.AdminUtilities$class.changeConfigs(AdminUtils.scala:52)
> at kafka.admin.AdminUtils$.changeConfigs(AdminUtils.scala:63)
> at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:103)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:70)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6050) --entity-name should print error message
[
https://issues.apache.org/jira/browse/KAFKA-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom Bentley updated KAFKA-6050:
---
Summary: --entity-name should print error message (was: Cannot
alter default topic config)
> --entity-name should print error message
> --
>
> Key: KAFKA-6050
> URL: https://issues.apache.org/jira/browse/KAFKA-6050
> Project: Kafka
> Issue Type: Bug
>Reporter: Tom Bentley
>
> The command to describe the default topic config
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 \
> --describe --entity-type topics --entity-name ''
> {noformat}
> returns without error, but the equivalent command to alter the default topic
> config:
> {noformat}
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter \
> --entity-type topics --entity-name '' --add-config
> retention.ms=1000
> {noformat}
> returns an error:
> {noformat}
> Error while executing config command Topic name "" is illegal, it
> contains a character other than ASCII alphanumerics, '.', '_' and '-'
> org.apache.kafka.common.errors.InvalidTopicException: Topic name ""
> is illegal, it contains a character other than ASCII alphanumerics, '.', '_'
> and '-'
> at org.apache.kafka.common.internals.Topic.validate(Topic.java:45)
> at kafka.admin.AdminUtils$.validateTopicConfig(AdminUtils.scala:578)
> at kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:595)
> at kafka.admin.AdminUtilities$class.changeConfigs(AdminUtils.scala:52)
> at kafka.admin.AdminUtils$.changeConfigs(AdminUtils.scala:63)
> at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:103)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:70)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6032) Unit Tests should be independent of locale settings
[
https://issues.apache.org/jira/browse/KAFKA-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201005#comment-16201005
]
ASF GitHub Bot commented on KAFKA-6032:
---
GitHub user gilles-degols opened a pull request:
https://github.com/apache/kafka/pull/4061
KAFKA-6032: Unit Tests should be independent of locale settings
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gilles-degols/kafka
KAFKA-6032-Unit-Tests-should-be-independent-of-locale-settings
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/4061.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4061
commit d80207ec7a74cf5cffacdf2cf9cd7d43f60880a5
Author: Gilles Degols
Date: 2017-10-11T21:30:06Z
Unit Tests should be independent of locale settings
> Unit Tests should be independent of locale settings
> ---
>
> Key: KAFKA-6032
> URL: https://issues.apache.org/jira/browse/KAFKA-6032
> Project: Kafka
> Issue Type: Bug
> Components: unit tests
>Affects Versions: 0.11.0.0
> Environment: Centos 7, Windows 10
> Locale = "fr_BE.UTF-8"
>Reporter: Gilles Degols
>Assignee: Gilles Degols
>Priority: Minor
> Labels: newbie
>
> If the system has locale settings like "fr_BE.UTF-8", 3 unit tests will fail:
> 1. org.apache.kafka.common.utils.ShellTest > testRunProgramWithErrorReturn
> 2. org.apache.kafka.common.utils.ShellTest >
> testAttemptToRunNonExistentProgram
> 3. org.apache.kafka.common.utils.UtilsTest > testFormatBytes
> They rely on string comparisons which will not work if the system throws an
> error in another language, or if the float format is different ("," instead
> of ".").
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6014) new consumer mirror maker halts after committing offsets to a deleted topic
[
https://issues.apache.org/jira/browse/KAFKA-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201060#comment-16201060
]
Gilles Degols commented on KAFKA-6014:
--
I would go for option 1, using a specific Exception to handle things correctly.
Is there a problem to use the InvalidTopicException or do you think a more
descriptive Exception would be needed?
> new consumer mirror maker halts after committing offsets to a deleted topic
> ---
>
> Key: KAFKA-6014
> URL: https://issues.apache.org/jira/browse/KAFKA-6014
> Project: Kafka
> Issue Type: Bug
>Reporter: Onur Karaman
>
> New consumer throws an unexpected KafkaException when trying to commit to a
> topic that has been deleted. MirrorMaker.commitOffsets doesn't attempt to
> catch the KafkaException and just kills the process. We didn't see this in
> the old consumer because old consumer just silently drops failed offset
> commits.
> I ran a quick experiment locally to prove the behavior. The experiment:
> 1. start up a single broker
> 2. create a single-partition topic t
> 3. create a new consumer that consumes topic t
> 4. make the consumer commit every few seconds
> 5. delete topic t
> 6. expect: KafkaException that kills the process.
> Here's my script:
> {code}
> package org.apache.kafka.clients.consumer;
> import org.apache.kafka.common.TopicPartition;
> import java.util.Collections;
> import java.util.List;
> import java.util.Properties;
> public class OffsetCommitTopicDeletionTest {
> public static void main(String[] args) throws InterruptedException {
> Properties props = new Properties();
> props.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
> "localhost:9090");
> props.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "g");
> props.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
> "org.apache.kafka.common.serialization.ByteArrayDeserializer");
> props.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
> "org.apache.kafka.common.serialization.ByteArrayDeserializer");
> props.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
> KafkaConsumer kafkaConsumer = new
> KafkaConsumer<>(props);
> TopicPartition partition = new TopicPartition("t", 0);
> List partitions =
> Collections.singletonList(partition);
> kafkaConsumer.assign(partitions);
> while (true) {
> kafkaConsumer.commitSync(Collections.singletonMap(partition, new
> OffsetAndMetadata(0, "")));
> Thread.sleep(1000);
> }
> }
> }
> {code}
> Here are the other commands:
> {code}
> > rm -rf /tmp/zookeeper/ /tmp/kafka-logs* logs*
> > ./gradlew clean jar
> > ./bin/zookeeper-server-start.sh config/zookeeper.properties
> > export LOG_DIR=logs0 && ./bin/kafka-server-start.sh
> > config/server0.properties
> > ./bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic t
> > --partitions 1 --replication-factor 1
> > ./bin/kafka-run-class.sh
> > org.apache.kafka.clients.consumer.OffsetCommitTopicDeletionTest
> > ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic t
> {code}
> Here is the output:
> {code}
> [2017-10-04 20:00:14,451] ERROR [Consumer clientId=consumer-1, groupId=g]
> Offset commit failed on partition t-0 at offset 0: This server does not host
> this topic-partition.
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> Exception in thread "main" org.apache.kafka.common.KafkaException: Partition
> t-0 may not exist or user may not have Describe access to topic
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:789)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:734)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:808)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:788)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
> at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:506)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)
> at
> org.apache.kafka.clients.consumer.internals.Consu
[jira] [Created] (KAFKA-6055) Running tools on Windows fail due to misconfigured JVM config
Vahid Hashemian created KAFKA-6055:
--
Summary: Running tools on Windows fail due to misconfigured JVM
config
Key: KAFKA-6055
URL: https://issues.apache.org/jira/browse/KAFKA-6055
Project: Kafka
Issue Type: Bug
Components: tools
Reporter: Vahid Hashemian
Assignee: Vahid Hashemian
Priority: Blocker
Fix For: 1.0.0
This affects the current trunk and 1.0.0 RC0.
When running any of the Windows commands under {{bin/windows}} the following
error is returned:
{code}
Missing +/- setting for VM option 'ExplicitGCInvokesConcurrent'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6055) Running tools on Windows fail due to misconfigured JVM config
[
https://issues.apache.org/jira/browse/KAFKA-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201126#comment-16201126
]
ASF GitHub Bot commented on KAFKA-6055:
---
GitHub user vahidhashemian opened a pull request:
https://github.com/apache/kafka/pull/4062
KAFKA-6055: Fix a JVM misconfiguration that affects Windows tools
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vahidhashemian/kafka KAFKA-6055
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/4062.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4062
commit 7954c3db11fb2547ba66ba177de99393bdaf4fb6
Author: Vahid Hashemian
Date: 2017-10-11T22:52:14Z
KAFKA-6055: Fix a JVM misconfiguration that affects Windows tools
> Running tools on Windows fail due to misconfigured JVM config
> -
>
> Key: KAFKA-6055
> URL: https://issues.apache.org/jira/browse/KAFKA-6055
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Blocker
> Fix For: 1.0.0
>
>
> This affects the current trunk and 1.0.0 RC0.
> When running any of the Windows commands under {{bin/windows}} the following
> error is returned:
> {code}
> Missing +/- setting for VM option 'ExplicitGCInvokesConcurrent'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6055) Running tools on Windows fail due to incorrect JVM config
[
https://issues.apache.org/jira/browse/KAFKA-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vahid Hashemian updated KAFKA-6055:
---
Summary: Running tools on Windows fail due to incorrect JVM config (was:
Running tools on Windows fail due to misconfigured JVM config)
> Running tools on Windows fail due to incorrect JVM config
> -
>
> Key: KAFKA-6055
> URL: https://issues.apache.org/jira/browse/KAFKA-6055
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Blocker
> Fix For: 1.0.0
>
>
> This affects the current trunk and 1.0.0 RC0.
> When running any of the Windows commands under {{bin/windows}} the following
> error is returned:
> {code}
> Missing +/- setting for VM option 'ExplicitGCInvokesConcurrent'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6055) Running tools on Windows fail due to incorrect JVM config
[
https://issues.apache.org/jira/browse/KAFKA-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vahid Hashemian updated KAFKA-6055:
---
Description:
This affects the current trunk and 1.0.0 RC0.
When running any of the Windows commands under {{bin/windows}} the following
error is returned:
{code}
Missing +/- setting for VM option 'ExplicitGCInvokesConcurrent'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
This error points to this JVM configuration in
{{bin\windows\kafka-run-class.bat}}: {{-XX:ExplicitGCInvokesConcurrent}}
The correct config is {{-XX:+ExplicitGCInvokesConcurrent}}
was:
This affects the current trunk and 1.0.0 RC0.
When running any of the Windows commands under {{bin/windows}} the following
error is returned:
{code}
Missing +/- setting for VM option 'ExplicitGCInvokesConcurrent'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
> Running tools on Windows fail due to incorrect JVM config
> -
>
> Key: KAFKA-6055
> URL: https://issues.apache.org/jira/browse/KAFKA-6055
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Blocker
> Fix For: 1.0.0
>
>
> This affects the current trunk and 1.0.0 RC0.
> When running any of the Windows commands under {{bin/windows}} the following
> error is returned:
> {code}
> Missing +/- setting for VM option 'ExplicitGCInvokesConcurrent'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> This error points to this JVM configuration in
> {{bin\windows\kafka-run-class.bat}}: {{-XX:ExplicitGCInvokesConcurrent}}
> The correct config is {{-XX:+ExplicitGCInvokesConcurrent}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6055) Running tools on Windows fail due to typo in JVM config
[
https://issues.apache.org/jira/browse/KAFKA-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vahid Hashemian updated KAFKA-6055:
---
Summary: Running tools on Windows fail due to typo in JVM config (was:
Running tools on Windows fail due to incorrect JVM config)
> Running tools on Windows fail due to typo in JVM config
> ---
>
> Key: KAFKA-6055
> URL: https://issues.apache.org/jira/browse/KAFKA-6055
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Blocker
> Fix For: 1.0.0
>
>
> This affects the current trunk and 1.0.0 RC0.
> When running any of the Windows commands under {{bin/windows}} the following
> error is returned:
> {code}
> Missing +/- setting for VM option 'ExplicitGCInvokesConcurrent'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> This error points to this JVM configuration in
> {{bin\windows\kafka-run-class.bat}}: {{-XX:ExplicitGCInvokesConcurrent}}
> The correct config is {{-XX:+ExplicitGCInvokesConcurrent}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KAFKA-5988) Consider removing StreamThread#STREAM_THREAD_ID_SEQUENCE
[ https://issues.apache.org/jira/browse/KAFKA-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-5988. --- Resolution: Won't Fix > Consider removing StreamThread#STREAM_THREAD_ID_SEQUENCE > > > Key: KAFKA-5988 > URL: https://issues.apache.org/jira/browse/KAFKA-5988 > Project: Kafka > Issue Type: Improvement >Reporter: Ted Yu >Assignee: siva santhalingam >Priority: Minor > > StreamThread#STREAM_THREAD_ID_SEQUENCE is used for naming (numbering) > StreamThread's . > It is used in create() which is called from a loop in KafkaStreams ctor. > We can remove STREAM_THREAD_ID_SEQUENCE and pass the loop index to create() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5863) Potential null dereference in DistributedHerder#reconfigureConnector()
[
https://issues.apache.org/jira/browse/KAFKA-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KAFKA-5863:
--
Description:
Here is the call chain:
{code}
RestServer.httpRequest(reconfigUrl, "POST",
taskProps, null);
{code}
In httpRequest():
{code}
} else if (responseCode >= 200 && responseCode < 300) {
InputStream is = connection.getInputStream();
T result = JSON_SERDE.readValue(is, responseFormat);
{code}
For readValue():
{code}
public T readValue(InputStream src, TypeReference valueTypeRef)
throws IOException, JsonParseException, JsonMappingException
{
return (T) _readMapAndClose(_jsonFactory.createParser(src),
_typeFactory.constructType(valueTypeRef));
{code}
Then there would be NPE in constructType():
{code}
public JavaType constructType(TypeReference typeRef)
{
// 19-Oct-2015, tatu: Simpler variant like so should work
return _fromAny(null, typeRef.getType(), EMPTY_BINDINGS);
{code}
was:
Here is the call chain:
{code}
RestServer.httpRequest(reconfigUrl, "POST",
taskProps, null);
{code}
In httpRequest():
{code}
} else if (responseCode >= 200 && responseCode < 300) {
InputStream is = connection.getInputStream();
T result = JSON_SERDE.readValue(is, responseFormat);
{code}
For readValue():
{code}
public T readValue(InputStream src, TypeReference valueTypeRef)
throws IOException, JsonParseException, JsonMappingException
{
return (T) _readMapAndClose(_jsonFactory.createParser(src),
_typeFactory.constructType(valueTypeRef));
{code}
Then there would be NPE in constructType():
{code}
public JavaType constructType(TypeReference typeRef)
{
// 19-Oct-2015, tatu: Simpler variant like so should work
return _fromAny(null, typeRef.getType(), EMPTY_BINDINGS);
{code}
> Potential null dereference in DistributedHerder#reconfigureConnector()
> --
>
> Key: KAFKA-5863
> URL: https://issues.apache.org/jira/browse/KAFKA-5863
> Project: Kafka
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> Here is the call chain:
> {code}
> RestServer.httpRequest(reconfigUrl, "POST",
> taskProps, null);
> {code}
> In httpRequest():
> {code}
> } else if (responseCode >= 200 && responseCode < 300) {
> InputStream is = connection.getInputStream();
> T result = JSON_SERDE.readValue(is, responseFormat);
> {code}
> For readValue():
> {code}
> public T readValue(InputStream src, TypeReference valueTypeRef)
> throws IOException, JsonParseException, JsonMappingException
> {
> return (T) _readMapAndClose(_jsonFactory.createParser(src),
> _typeFactory.constructType(valueTypeRef));
> {code}
> Then there would be NPE in constructType():
> {code}
> public JavaType constructType(TypeReference typeRef)
> {
> // 19-Oct-2015, tatu: Simpler variant like so should work
> return _fromAny(null, typeRef.getType(), EMPTY_BINDINGS);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6056) LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ offset range
[ https://issues.apache.org/jira/browse/KAFKA-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Filipiak updated KAFKA-6056: Summary: LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ offset range (was: LogCleaner always cleaning into 1 Segment might exeed relativ offset range) > LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ > offset range > > > Key: KAFKA-6056 > URL: https://issues.apache.org/jira/browse/KAFKA-6056 > Project: Kafka > Issue Type: Bug > Components: core, log >Affects Versions: 0.11.0.0 >Reporter: Jan Filipiak >Priority: Minor > > After having an Issue with compaction stopping for some time. It can be an > issue that the LogCleaner will always clean into 1 Segment per sizegroup. > Usually the Log enforces a maximum distance between min and max offset in a > LogSegment. If that Distance would be exeeded in maybeRoll() a new logsegment > would be rolled. I assume this is because relative offset might be stored as > integer. The LogCleaner OTOH is not going to roll a new LogSegment as its > only ever using 1 Segment to clean into. > A lenghty discussion about this can be found in the slack community: > https://confluentcommunity.slack.com/archives/C49R61XMM/p150691444105 > The observed stacktrace is as follows: > https://gist.github.com/brettrann/ce52343692696a45d5b9f4df723bcd14 > I could imagin also enfocing Integer.MAX_VALUE as offset distance in > groupSegmentsBySize in the LogCleaner to make sure a Segment doesnt exeed > this limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6056) LogCleaner always cleaning into 1 Segment might exeed relativ offset range
Jan Filipiak created KAFKA-6056: --- Summary: LogCleaner always cleaning into 1 Segment might exeed relativ offset range Key: KAFKA-6056 URL: https://issues.apache.org/jira/browse/KAFKA-6056 Project: Kafka Issue Type: Bug Components: core, log Affects Versions: 0.11.0.0 Reporter: Jan Filipiak Priority: Minor After having an Issue with compaction stopping for some time. It can be an issue that the LogCleaner will always clean into 1 Segment per sizegroup. Usually the Log enforces a maximum distance between min and max offset in a LogSegment. If that Distance would be exeeded in maybeRoll() a new logsegment would be rolled. I assume this is because relative offset might be stored as integer. The LogCleaner OTOH is not going to roll a new LogSegment as its only ever using 1 Segment to clean into. A lenghty discussion about this can be found in the slack community: https://confluentcommunity.slack.com/archives/C49R61XMM/p150691444105 The observed stacktrace is as follows: https://gist.github.com/brettrann/ce52343692696a45d5b9f4df723bcd14 I could imagin also enfocing Integer.MAX_VALUE as offset distance in groupSegmentsBySize in the LogCleaner to make sure a Segment doesnt exeed this limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6056) LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ offset range
[ https://issues.apache.org/jira/browse/KAFKA-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201439#comment-16201439 ] Brett Rann commented on KAFKA-6056: --- just popping my name in. this error happened to me, and I discussed it with Jan in slack. He seems to understand it better than I do so raised this ticket. > LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ > offset range > > > Key: KAFKA-6056 > URL: https://issues.apache.org/jira/browse/KAFKA-6056 > Project: Kafka > Issue Type: Bug > Components: core, log >Affects Versions: 0.11.0.0 >Reporter: Jan Filipiak >Priority: Minor > > After having an Issue with compaction stopping for some time. It can be an > issue that the LogCleaner will always clean into 1 Segment per sizegroup. > Usually the Log enforces a maximum distance between min and max offset in a > LogSegment. If that Distance would be exeeded in maybeRoll() a new logsegment > would be rolled. I assume this is because relative offset might be stored as > integer. The LogCleaner OTOH is not going to roll a new LogSegment as its > only ever using 1 Segment to clean into. > A lenghty discussion about this can be found in the slack community: > https://confluentcommunity.slack.com/archives/C49R61XMM/p150691444105 > The observed stacktrace is as follows: > https://gist.github.com/brettrann/ce52343692696a45d5b9f4df723bcd14 > I could imagin also enfocing Integer.MAX_VALUE as offset distance in > groupSegmentsBySize in the LogCleaner to make sure a Segment doesnt exeed > this limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6056) LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ offset range
[ https://issues.apache.org/jira/browse/KAFKA-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201451#comment-16201451 ] Jan Filipiak commented on KAFKA-6056: - Besides not mentioned in the comment, the code actually checks this condition. since here https://issues.apache.org/jira/browse/KAFKA-4451 So it can only affect users upgrading from before 10.2 that do have a LogSegment where the range is already exeeded inside 1 segment > LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ > offset range > > > Key: KAFKA-6056 > URL: https://issues.apache.org/jira/browse/KAFKA-6056 > Project: Kafka > Issue Type: Bug > Components: core, log >Affects Versions: 0.11.0.0 >Reporter: Jan Filipiak >Priority: Minor > > After having an Issue with compaction stopping for some time. It can be an > issue that the LogCleaner will always clean into 1 Segment per sizegroup. > Usually the Log enforces a maximum distance between min and max offset in a > LogSegment. If that Distance would be exeeded in maybeRoll() a new logsegment > would be rolled. I assume this is because relative offset might be stored as > integer. The LogCleaner OTOH is not going to roll a new LogSegment as its > only ever using 1 Segment to clean into. > A lenghty discussion about this can be found in the slack community: > https://confluentcommunity.slack.com/archives/C49R61XMM/p150691444105 > The observed stacktrace is as follows: > https://gist.github.com/brettrann/ce52343692696a45d5b9f4df723bcd14 > I could imagin also enfocing Integer.MAX_VALUE as offset distance in > groupSegmentsBySize in the LogCleaner to make sure a Segment doesnt exeed > this limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6056) LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ offset range
[ https://issues.apache.org/jira/browse/KAFKA-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201452#comment-16201452 ] Brett Rann commented on KAFKA-6056: --- That was the case for me. The upgrade was from 0.9.0.1 -> 0.11.0.1 > LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ > offset range > > > Key: KAFKA-6056 > URL: https://issues.apache.org/jira/browse/KAFKA-6056 > Project: Kafka > Issue Type: Bug > Components: core, log >Affects Versions: 0.11.0.0 >Reporter: Jan Filipiak >Priority: Minor > > After having an Issue with compaction stopping for some time. It can be an > issue that the LogCleaner will always clean into 1 Segment per sizegroup. > Usually the Log enforces a maximum distance between min and max offset in a > LogSegment. If that Distance would be exeeded in maybeRoll() a new logsegment > would be rolled. I assume this is because relative offset might be stored as > integer. The LogCleaner OTOH is not going to roll a new LogSegment as its > only ever using 1 Segment to clean into. > A lenghty discussion about this can be found in the slack community: > https://confluentcommunity.slack.com/archives/C49R61XMM/p150691444105 > The observed stacktrace is as follows: > https://gist.github.com/brettrann/ce52343692696a45d5b9f4df723bcd14 > I could imagin also enfocing Integer.MAX_VALUE as offset distance in > groupSegmentsBySize in the LogCleaner to make sure a Segment doesnt exeed > this limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6056) LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ offset range
[ https://issues.apache.org/jira/browse/KAFKA-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201455#comment-16201455 ] Jan Filipiak commented on KAFKA-6056: - A valid workaround for partitions containing the segments should be to replicate them to a broker of a version >10.2 wich will then roll the segment while building up the replica. As there is no problem in the code actually. One could think of including this into the upgrade notices? > LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ > offset range > > > Key: KAFKA-6056 > URL: https://issues.apache.org/jira/browse/KAFKA-6056 > Project: Kafka > Issue Type: Bug > Components: core, log >Affects Versions: 0.11.0.0 >Reporter: Jan Filipiak >Priority: Minor > > After having an Issue with compaction stopping for some time. It can be an > issue that the LogCleaner will always clean into 1 Segment per sizegroup. > Usually the Log enforces a maximum distance between min and max offset in a > LogSegment. If that Distance would be exeeded in maybeRoll() a new logsegment > would be rolled. I assume this is because relative offset might be stored as > integer. The LogCleaner OTOH is not going to roll a new LogSegment as its > only ever using 1 Segment to clean into. > A lenghty discussion about this can be found in the slack community: > https://confluentcommunity.slack.com/archives/C49R61XMM/p150691444105 > The observed stacktrace is as follows: > https://gist.github.com/brettrann/ce52343692696a45d5b9f4df723bcd14 > I could imagin also enfocing Integer.MAX_VALUE as offset distance in > groupSegmentsBySize in the LogCleaner to make sure a Segment doesnt exeed > this limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-6056) LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ offset range
[ https://issues.apache.org/jira/browse/KAFKA-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201480#comment-16201480 ] Brett Rann commented on KAFKA-6056: --- That workaround indeed seems to work. I moved it to a different 0.11.0.1 broker and the problem went away. the reason it existed on an 0.11.0.1 broker in the first place was because it inherited from a 0.9.0.1 upgrade. > LogCleaner always cleaning into 1 Segment per sizegroup might exeed relativ > offset range > > > Key: KAFKA-6056 > URL: https://issues.apache.org/jira/browse/KAFKA-6056 > Project: Kafka > Issue Type: Bug > Components: core, log >Affects Versions: 0.11.0.0 >Reporter: Jan Filipiak >Priority: Minor > > After having an Issue with compaction stopping for some time. It can be an > issue that the LogCleaner will always clean into 1 Segment per sizegroup. > Usually the Log enforces a maximum distance between min and max offset in a > LogSegment. If that Distance would be exeeded in maybeRoll() a new logsegment > would be rolled. I assume this is because relative offset might be stored as > integer. The LogCleaner OTOH is not going to roll a new LogSegment as its > only ever using 1 Segment to clean into. > A lenghty discussion about this can be found in the slack community: > https://confluentcommunity.slack.com/archives/C49R61XMM/p150691444105 > The observed stacktrace is as follows: > https://gist.github.com/brettrann/ce52343692696a45d5b9f4df723bcd14 > I could imagin also enfocing Integer.MAX_VALUE as offset distance in > groupSegmentsBySize in the LogCleaner to make sure a Segment doesnt exeed > this limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-3584) NullPointer Exception found when Delete Topic and Log delete concurrent
[ https://issues.apache.org/jira/browse/KAFKA-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201525#comment-16201525 ] wangchen commented on KAFKA-3584: - i am find! > NullPointer Exception found when Delete Topic and Log delete concurrent > --- > > Key: KAFKA-3584 > URL: https://issues.apache.org/jira/browse/KAFKA-3584 > Project: Kafka > Issue Type: Bug > Components: core, log >Affects Versions: 0.9.0.1 >Reporter: Pengwei >Assignee: Manikumar >Priority: Minor > Fix For: 0.10.0.0 > > > [2016-03-19 17:23:45,760] ERROR Uncaught exception in scheduled task > ‘kafka-log-retention’ (kafka.utils.KafkaScheduler) > java.lang.NullPointerException > at kafka.log.Log.activeSegment(Log.scala:824) > at kafka.log.Log.deleteOldSegments(Log.scala:602) > at kafka.log.LogManager.kafka$log$LogManager -- This message was sent by Atlassian JIRA (v6.4.14#64029)
