[jira] [Assigned] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huxihx reassigned KAFKA-5724: - Assignee: huxihx > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122966#comment-16122966 ] ASF GitHub Bot commented on KAFKA-5724: --- GitHub user huxihx opened a pull request: https://github.com/apache/kafka/pull/3655 KAFKA-5724: AbstractPartitionAssignor should support assignment for topics with non-consecutive partitions Current design does consider the siutation when user creates a topic via KafkaAdminClient whose partitions are not consecutive or zero-based. In such case, consumer does not work since assignor failed to assign partitions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huxihx/kafka KAFKA-5724 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3655.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3655 commit 9ffa5a8961039affcc829b09912584b56063ddf3 Author: huxihx Date: 2017-08-11T07:20:52Z KAFKA-5724: AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero Current design does consider the siutation when user creates a topic via KafkaAdminClient whose partitions are not consecutive or zero-based. In such case, consumer does not work since assignor failed to assign partitions. > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5714) Allow whitespaces in the principal name
[
https://issues.apache.org/jira/browse/KAFKA-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123074#comment-16123074
]
Manikumar commented on KAFKA-5714:
--
[~a...@confluent.io] From the code, I can see that we are allowing white spaces
in the principal name. Can you explain more about your observation/scenario?
> Allow whitespaces in the principal name
> ---
>
> Key: KAFKA-5714
> URL: https://issues.apache.org/jira/browse/KAFKA-5714
> Project: Kafka
> Issue Type: Improvement
> Components: security
>Affects Versions: 0.10.2.1
>Reporter: Alla Tumarkin
>
> Request
> Improve parser behavior to allow whitespaces in the principal name in the
> config file, as in:
> {code}
> super.users=User:CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown
> {code}
> Background
> Current implementation requires that there are no whitespaces after commas,
> i.e.
> {code}
> super.users=User:CN=Unknown,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown
> {code}
> Note: having a semicolon at the end doesn't help, i.e. this does not work
> either
> {code}
> super.users=User:CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (KAFKA-5714) Allow whitespaces in the principal name
[
https://issues.apache.org/jira/browse/KAFKA-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar reassigned KAFKA-5714:
Assignee: Manikumar
> Allow whitespaces in the principal name
> ---
>
> Key: KAFKA-5714
> URL: https://issues.apache.org/jira/browse/KAFKA-5714
> Project: Kafka
> Issue Type: Improvement
> Components: security
>Affects Versions: 0.10.2.1
>Reporter: Alla Tumarkin
>Assignee: Manikumar
>
> Request
> Improve parser behavior to allow whitespaces in the principal name in the
> config file, as in:
> {code}
> super.users=User:CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown
> {code}
> Background
> Current implementation requires that there are no whitespaces after commas,
> i.e.
> {code}
> super.users=User:CN=Unknown,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown
> {code}
> Note: having a semicolon at the end doesn't help, i.e. this does not work
> either
> {code}
> super.users=User:CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5637) Document compatibility and release policies
[ https://issues.apache.org/jira/browse/KAFKA-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123181#comment-16123181 ] Sönke Liebau commented on KAFKA-5637: - My personal expectation for a stable, public API would be that it is binary as well as source compatible i.e. no changes at all to anything public facing. Verifying this is probably not an easy task, there are tools out there that analyse compatibility at both levels, however embedding these in the build process and getting it to recognize our stability annotations as well as the rules in relation to version changes seems like a quite daunting task to me tbh. I suspect that this is something we need to catch when reviewing changes (can be tool assisted for larger changes - but as a manual run). Additionally we could add a manual step to the release process to generate a compatibility report against previous versions and evaluate this against stable APIs and the specific rules in effect for the version change. As for different rules for unstable APIs, I'd say that we stick to one definition of "compatible" across all levels of stability. > Document compatibility and release policies > --- > > Key: KAFKA-5637 > URL: https://issues.apache.org/jira/browse/KAFKA-5637 > Project: Kafka > Issue Type: Improvement > Components: documentation >Reporter: Ismael Juma >Assignee: Sönke Liebau > Fix For: 1.0.0 > > > We should document our compatibility and release policies in one place so > that people have the correct expectations. This is generally important, but > more so now that we are releasing 1.0.0. > I extracted the following topics from the mailing list thread as the ones > that should be documented as a minimum: > *Code stability* > * Explanation of stability annotations and their implications > * Explanation of what public apis are > * *Discussion point: * Do we want to keep the _unstable_ annotation or is > _evolving_ sufficient going forward? > *Support duration* > * How long are versions supported? > * How far are bugfixes backported? > * How far are security fixes backported? > * How long are protocol versions supported by subsequent code versions? > * How long are older clients supported? > * How long are older brokers supported? > I will create an initial pull request to add a section to the documentation > as basis for further discussion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5702) Refactor StreamThread to separate concerns and enable better testability
[
https://issues.apache.org/jira/browse/KAFKA-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123199#comment-16123199
]
ASF GitHub Bot commented on KAFKA-5702:
---
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/3624
> Refactor StreamThread to separate concerns and enable better testability
>
>
> Key: KAFKA-5702
> URL: https://issues.apache.org/jira/browse/KAFKA-5702
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Damian Guy
>Assignee: Damian Guy
>
> {{StreamThread}} does a lot of stuff, i.e., managing and creating tasks,
> getting data from consumers, updating standby tasks, punctuating, rebalancing
> etc. With the current design it is extremely hard to reason about and is
> quite tightly coupled.
> We need to start to tease out some of the separate concerns from
> StreamThread, ie, TaskManager, RebalanceListener etc.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5676) MockStreamsMetrics should be in o.a.k.test
[
https://issues.apache.org/jira/browse/KAFKA-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120968#comment-16120968
]
Chanchal Singh edited comment on KAFKA-5676 at 8/11/17 12:22 PM:
-
also request you to let me know if there is any faster channel for
communication other then jira. Created merge request but showing conflict of
code changes with trunk . please look into it
was (Author: chanchal.kafka):
also request you to let me know if there is any faster channel for
communication other then jira.
> MockStreamsMetrics should be in o.a.k.test
> --
>
> Key: KAFKA-5676
> URL: https://issues.apache.org/jira/browse/KAFKA-5676
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Assignee: Chanchal Singh
> Labels: newbie
>
> {{MockStreamsMetrics}}'s package should be `o.a.k.test` not
> `o.a.k.streams.processor.internals`.
> In addition, it should not require a {{Metrics}} parameter in its constructor
> as it is only needed for its extended base class; the right way of mocking
> should be implementing {{StreamsMetrics}} with mock behavior than extended a
> real implementaion of {{StreamsMetricsImpl}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-3268) Refactor existing CLI scripts to use KafkaAdminClient
[ https://issues.apache.org/jira/browse/KAFKA-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123279#comment-16123279 ] Viktor Somogyi commented on KAFKA-3268: --- [~ppatierno] sure, it'd be great to sync on the solution/interfaces, basic utilities and behavior. I'll review your changes. > Refactor existing CLI scripts to use KafkaAdminClient > - > > Key: KAFKA-3268 > URL: https://issues.apache.org/jira/browse/KAFKA-3268 > Project: Kafka > Issue Type: Sub-task >Reporter: Grant Henke >Assignee: Viktor Somogyi > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5723) Refactor BrokerApiVersionsCommand to use the new AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi updated KAFKA-5723: -- Summary: Refactor BrokerApiVersionsCommand to use the new AdminClient (was: Refactor BrokerApiVersionsCommand to use the KafkaAdminClient) > Refactor BrokerApiVersionsCommand to use the new AdminClient > > > Key: KAFKA-5723 > URL: https://issues.apache.org/jira/browse/KAFKA-5723 > Project: Kafka > Issue Type: Improvement >Reporter: Viktor Somogyi >Assignee: Viktor Somogyi > > Currently it uses the deprecated AdminClient and in order to remove usages of > that client, this class needs to be refactored. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5722) Refactor ConfigCommand to use the new AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi updated KAFKA-5722: -- Summary: Refactor ConfigCommand to use the new AdminClient (was: Refactor ConfigCommand to use the KafkaAdminClient) > Refactor ConfigCommand to use the new AdminClient > - > > Key: KAFKA-5722 > URL: https://issues.apache.org/jira/browse/KAFKA-5722 > Project: Kafka > Issue Type: Improvement >Reporter: Viktor Somogyi >Assignee: Viktor Somogyi > Labels: kip, needs-kip > > The ConfigCommand currently uses a direct connection to zookeeper. The > zookeeper dependency should be deprecated and an AdminClient API created to > be used instead. > This change requires a KIP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5725) Additional failure testing for streams with bouncing brokers
Eno Thereska created KAFKA-5725: --- Summary: Additional failure testing for streams with bouncing brokers Key: KAFKA-5725 URL: https://issues.apache.org/jira/browse/KAFKA-5725 Project: Kafka Issue Type: Test Components: streams Affects Versions: 0.11.0.0 Reporter: Eno Thereska Assignee: Eno Thereska Fix For: 0.11.0.1 We have tests/streams/streams_broker_bounce_test.py that tests streams' robustness when some brokers quit or are terminated. We do not have coverage for transient failures, such as when brokers come back. Add such tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-4643) Improve test coverage of StreamsKafkaClient
[
https://issues.apache.org/jira/browse/KAFKA-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123434#comment-16123434
]
ASF GitHub Bot commented on KAFKA-4643:
---
GitHub user adyach opened a pull request:
https://github.com/apache/kafka/pull/3659
KAFKA-4643: Improve test coverage of StreamsKafkaClient
The commit brings improved test coverage for StreamsKafkaClientTest.java
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/adyach/kafka trunk
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3659.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3659
commit 2ef77edcd853e23572b9a0543af2907ef89bd241
Author: Andrey Dyachkov
Date: 2017-03-10T21:51:12Z
KAFKA-4657: improved test coverage
This commmit brings improved test coverage for window store fetch method
and WindowStoreIterator
commit 3a386b350415baaf141b181222bce3397750adea
Author: Andrey Dyachkov
Date: 2017-03-11T13:58:47Z
KAFKA-4657: split test
commit 90e68164c69bcbc553f4eaadebffd2bac5eb39dd
Author: Andrey Dyachkov
Date: 2017-03-13T08:30:19Z
KAFKA-4657: fixed test names
commit 413ae41012a20443c22418ede96a9b6040814873
Author: Andrey Dyachkov
Date: 2017-03-13T19:55:24Z
Merge branch 'trunk' of github.com:apache/kafka into trunk
commit 6957d6ee542e34f9eb24f6bc5882fb104350fc84
Author: Andrey Dyachkov
Date: 2017-05-23T07:45:46Z
Merge remote-tracking branch 'upstream/trunk' into trunk
commit 1c14d1fda438d9c044b2e14676d655196b2c0832
Author: Andrey Dyachkov
Date: 2017-07-24T11:06:58Z
Merge branch 'trunk' of github.com:adyach/kafka into trunk
commit 7ac217510847acfeb714e0058bf3bcce1c214029
Author: Andrey Dyachkov
Date: 2017-07-24T11:07:31Z
Merge branch 'trunk' of github.com:apache/kafka into trunk
commit a4fd8a05e91f483b7cdcbbe4997c25e1f8253f83
Author: Andrey Dyachkov
Date: 2017-08-11T12:24:44Z
Merge branch 'trunk' of github.com:apache/kafka into trunk
commit 288b6f75ffc072bd20b9fc7972feeb10abdb13d9
Author: Andrey Dyachkov
Date: 2017-08-11T14:23:54Z
KAFKA-4643: exception test cases
The commit brings improved test coverage for StreamsKafkaClientTest.java
> Improve test coverage of StreamsKafkaClient
> ---
>
> Key: KAFKA-4643
> URL: https://issues.apache.org/jira/browse/KAFKA-4643
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Damian Guy
>
> Exception paths not tested.
> {{getTopicMetadata}} not tested
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-4643) Improve test coverage of StreamsKafkaClient
[
https://issues.apache.org/jira/browse/KAFKA-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123436#comment-16123436
]
Andrey Dyachkov commented on KAFKA-4643:
I added a couple of test case for exception flows. Please, check it out.
> Improve test coverage of StreamsKafkaClient
> ---
>
> Key: KAFKA-4643
> URL: https://issues.apache.org/jira/browse/KAFKA-4643
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Damian Guy
>
> Exception paths not tested.
> {{getTopicMetadata}} not tested
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5723) Refactor BrokerApiVersionsCommand to use the new AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123482#comment-16123482 ] Andrey Dyachkov commented on KAFKA-5723: Could I try to accomplish that task? It seems a lot smaller comparing to other from KAFKA-3268, and does not require a KIP? > Refactor BrokerApiVersionsCommand to use the new AdminClient > > > Key: KAFKA-5723 > URL: https://issues.apache.org/jira/browse/KAFKA-5723 > Project: Kafka > Issue Type: Improvement >Reporter: Viktor Somogyi >Assignee: Viktor Somogyi > > Currently it uses the deprecated AdminClient and in order to remove usages of > that client, this class needs to be refactored. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5726) KafkaConsumer.subscribe() override that takes just Pattern without ConsumerRebalanceListener
Yeva Byzek created KAFKA-5726:
-
Summary: KafkaConsumer.subscribe() override that takes just
Pattern without ConsumerRebalanceListener
Key: KAFKA-5726
URL: https://issues.apache.org/jira/browse/KAFKA-5726
Project: Kafka
Issue Type: Improvement
Components: consumer
Affects Versions: 0.11.0.0
Reporter: Yeva Byzek
Priority: Minor
Request: provide {{subscribe(Pattern pattern)}} override, similar to
{{subscribe(Collection topics)}},
Today, for a consumer to subscribe to topics based on a regular expression
(i.e. {{Pattern}}), the only method option also requires to pass in a
{{ConsumerRebalanceListener}}. This is not user-friendly to require this second
argument. It seems {{new NoOpConsumerRebalanceListener()}} has to be used.
Use case: multi datacenter, allowing easier subscription to multiple topics
prefixed with datacenter names, just by using a pattern subscription.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5726) KafkaConsumer.subscribe() overload that takes just Pattern without ConsumerRebalanceListener
[
https://issues.apache.org/jira/browse/KAFKA-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-5726:
---
Labels: needs-kip newbie usability (was: usability)
> KafkaConsumer.subscribe() overload that takes just Pattern without
> ConsumerRebalanceListener
>
>
> Key: KAFKA-5726
> URL: https://issues.apache.org/jira/browse/KAFKA-5726
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
>Affects Versions: 0.11.0.0
>Reporter: Yeva Byzek
>Priority: Minor
> Labels: needs-kip, newbie, usability
>
> Request: provide {{subscribe(Pattern pattern)}} overload, similar to
> {{subscribe(Collection topics)}},
> Today, for a consumer to subscribe to topics based on a regular expression
> (i.e. {{Pattern}}), the only method option also requires to pass in a
> {{ConsumerRebalanceListener}}. This is not user-friendly to require this
> second argument. It seems {{new NoOpConsumerRebalanceListener()}} has to be
> used.
> Use case: multi datacenter, allowing easier subscription to multiple topics
> prefixed with datacenter names, just by using a pattern subscription.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5726) KafkaConsumer.subscribe() overload that takes just Pattern without ConsumerRebalanceListener
[
https://issues.apache.org/jira/browse/KAFKA-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-5726:
---
Description:
Request: provide {{subscribe(Pattern pattern)}} overload, similar to
{{subscribe(Collection topics)}},
Today, for a consumer to subscribe to topics based on a regular expression
(i.e. {{Pattern}}), the only method option also requires to pass in a
{{ConsumerRebalanceListener}}. This is not user-friendly to require this second
argument. It seems {{new NoOpConsumerRebalanceListener()}} has to be used.
Use case: multi datacenter, allowing easier subscription to multiple topics
prefixed with datacenter names, just by using a pattern subscription.
was:
Request: provide {{subscribe(Pattern pattern)}} override, similar to
{{subscribe(Collection topics)}},
Today, for a consumer to subscribe to topics based on a regular expression
(i.e. {{Pattern}}), the only method option also requires to pass in a
{{ConsumerRebalanceListener}}. This is not user-friendly to require this second
argument. It seems {{new NoOpConsumerRebalanceListener()}} has to be used.
Use case: multi datacenter, allowing easier subscription to multiple topics
prefixed with datacenter names, just by using a pattern subscription.
> KafkaConsumer.subscribe() overload that takes just Pattern without
> ConsumerRebalanceListener
>
>
> Key: KAFKA-5726
> URL: https://issues.apache.org/jira/browse/KAFKA-5726
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
>Affects Versions: 0.11.0.0
>Reporter: Yeva Byzek
>Priority: Minor
> Labels: needs-kip, newbie, usability
>
> Request: provide {{subscribe(Pattern pattern)}} overload, similar to
> {{subscribe(Collection topics)}},
> Today, for a consumer to subscribe to topics based on a regular expression
> (i.e. {{Pattern}}), the only method option also requires to pass in a
> {{ConsumerRebalanceListener}}. This is not user-friendly to require this
> second argument. It seems {{new NoOpConsumerRebalanceListener()}} has to be
> used.
> Use case: multi datacenter, allowing easier subscription to multiple topics
> prefixed with datacenter names, just by using a pattern subscription.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5726) KafkaConsumer.subscribe() overload that takes just Pattern without ConsumerRebalanceListener
[
https://issues.apache.org/jira/browse/KAFKA-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-5726:
---
Summary: KafkaConsumer.subscribe() overload that takes just Pattern without
ConsumerRebalanceListener (was: KafkaConsumer.subscribe() override that takes
just Pattern without ConsumerRebalanceListener)
> KafkaConsumer.subscribe() overload that takes just Pattern without
> ConsumerRebalanceListener
>
>
> Key: KAFKA-5726
> URL: https://issues.apache.org/jira/browse/KAFKA-5726
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
>Affects Versions: 0.11.0.0
>Reporter: Yeva Byzek
>Priority: Minor
> Labels: needs-kip, newbie, usability
>
> Request: provide {{subscribe(Pattern pattern)}} override, similar to
> {{subscribe(Collection topics)}},
> Today, for a consumer to subscribe to topics based on a regular expression
> (i.e. {{Pattern}}), the only method option also requires to pass in a
> {{ConsumerRebalanceListener}}. This is not user-friendly to require this
> second argument. It seems {{new NoOpConsumerRebalanceListener()}} has to be
> used.
> Use case: multi datacenter, allowing easier subscription to multiple topics
> prefixed with datacenter names, just by using a pattern subscription.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5562) Do streams state directory cleanup on a single thread
[
https://issues.apache.org/jira/browse/KAFKA-5562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123503#comment-16123503
]
ASF GitHub Bot commented on KAFKA-5562:
---
Github user dguy closed the pull request at:
https://github.com/apache/kafka/pull/3654
> Do streams state directory cleanup on a single thread
> -
>
> Key: KAFKA-5562
> URL: https://issues.apache.org/jira/browse/KAFKA-5562
> Project: Kafka
> Issue Type: Bug
>Reporter: Damian Guy
>Assignee: Damian Guy
>
> Currently in streams we clean up old state directories every so often (as
> defined by {{state.cleanup.delay.ms}}). However, every StreamThread runs the
> cleanup, which is both unnecessary and can potentially lead to race
> conditions.
> It would be better to perform the state cleanup on a single thread and only
> when the {{KafkaStreams}} instance is in a running state.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123563#comment-16123563 ] Jason Gustafson commented on KAFKA-5724: [~allenxwang] I fear this is not the only place we make this assumption. Can you clarify specifically how an advanced tool creates a topic with a non-zero first partition? > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123584#comment-16123584 ] Allen Wang commented on KAFKA-5724: --- [~hachikuji] We have our own topic creation tool where we take advantage of AdminUtils.assignReplicasToBrokers() where you can pass a starting partition ID. The API will return an assignment which can be passed to TopicCommand or AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK() API. > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123631#comment-16123631 ] Allen Wang commented on KAFKA-5724: --- [~hachikuji] Please note that TopicCommand can take the replica assignment as an argument to create topic. So I believe it does not require much advanced tooling to get into this situation. > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-4682) Committed offsets should not be deleted if a consumer is still active
[ https://issues.apache.org/jira/browse/KAFKA-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123742#comment-16123742 ] Vahid Hashemian commented on KAFKA-4682: [~hachikuji] Thank you for your comments. You seem to be looking at this with an inclination to get rid of the retention time from the OffsetCommit protocol. I think with my comments below I'm considering the alternative: # Ewen's KIP proposes to increase the default retention from 1 day to 7 days. So, allowing consumers to set a lower timeout (for the console consumer) seems to be helpful after his KIP; the same way allowing them to set a higher timeout (for actual consumer applications) is helpful before his KIP. # Even if we have offset-level expiration, all offsets in the group should expire together, because the expiration timer starts ticking for all partitions at the same time (when the group becomes empty). The only exception is when a consumer has set a non-default retention time for particular partitions (e.g. using the OffsetCommit API). # Agreed. The expiration timestamp won't make sense. Perhaps the retention time should be stored and whether to expire or not could be calculated on the fly from the time group becomes empty + retention time (we would need to somehow keep the timestamp of the group becoming empty). This expiration check needs to be performed only if the group is empty; otherwise there is no need to expire at all. # I don't have a strong feeling about this. It's for sure simpler to let all offsets expire at the same time. And if we keep the individual offset retention it would be easier to change this in case the cache size becomes an issue. I think there is a risk involved in removing the individual retention from the protocol: could some requirement arise in the future that makes us bring it back to the protocol? One option is to let that field stay for now, and remove it later once we are more certain that it won't be needed back. > Committed offsets should not be deleted if a consumer is still active > - > > Key: KAFKA-4682 > URL: https://issues.apache.org/jira/browse/KAFKA-4682 > Project: Kafka > Issue Type: Bug >Reporter: James Cheng > > Kafka will delete committed offsets that are older than > offsets.retention.minutes > If there is an active consumer on a low traffic partition, it is possible > that Kafka will delete the committed offset for that consumer. Once the > offset is deleted, a restart or a rebalance of that consumer will cause the > consumer to not find any committed offset and start consuming from > earliest/latest (depending on auto.offset.reset). I'm not sure, but a broker > failover might also cause you to start reading from auto.offset.reset (due to > broker restart, or coordinator failover). > I think that Kafka should only delete offsets for inactive consumers. The > timer should only start after a consumer group goes inactive. For example, if > a consumer group goes inactive, then after 1 week, delete the offsets for > that consumer group. This is a solution that [~junrao] mentioned in > https://issues.apache.org/jira/browse/KAFKA-3806?focusedCommentId=15323521&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15323521 > The current workarounds are to: > # Commit an offset on every partition you own on a regular basis, making sure > that it is more frequent than offsets.retention.minutes (a broker-side > setting that a consumer might not be aware of) > or > # Turn the value of offsets.retention.minutes up really really high. You have > to make sure it is higher than any valid low-traffic rate that you want to > support. For example, if you want to support a topic where someone produces > once a month, you would have to set offsetes.retention.mintues to 1 month. > or > # Turn on enable.auto.commit (this is essentially #1, but easier to > implement). > None of these are ideal. > #1 can be spammy. It requires your consumers know something about how the > brokers are configured. Sometimes it is out of your control. Mirrormaker, for > example, only commits offsets on partitions where it receives data. And it is > duplication that you need to put into all of your consumers. > #2 has disk-space impact on the broker (in __consumer_offsets) as well as > memory-size on the broker (to answer OffsetFetch). > #3 I think has the potential for message loss (the consumer might commit on > messages that are not yet fully processed) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-4682) Committed offsets should not be deleted if a consumer is still active
[ https://issues.apache.org/jira/browse/KAFKA-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123742#comment-16123742 ] Vahid Hashemian edited comment on KAFKA-4682 at 8/11/17 5:56 PM: - [~hachikuji] Thank you for your comments. You seem to be looking at this with an inclination to get rid of the retention time from the OffsetCommit protocol. I think with my comments below I'm considering the alternative: # Ewen's KIP proposes to increase the default retention from 1 day to 7 days. So, allowing consumers to set a lower timeout (for the console consumer) seems to be helpful after his KIP; the same way allowing them to set a higher timeout (for actual consumer applications) is helpful before his KIP. # Even if we have offset-level expiration, all offsets in the group should expire together, because the expiration timer starts ticking for all partitions at the same time (when the group becomes empty). The only exception is when a consumer has set a non-default retention time for particular partitions (e.g. using the OffsetCommit API). # Agreed. The expiration timestamp won't make sense. Perhaps the retention time should be stored and whether to expire or not could be calculated on the fly from the time group becomes empty + retention time (we would need to somehow keep the timestamp of the group becoming empty). This expiration check needs to be performed only if the group is empty; otherwise there is no need to expire at all. # I don't have a strong feeling about this. It's for sure simpler to let all offsets expire at the same time. And if we keep the individual offset retention it would be easier to change this in case the cache size becomes an issue. I think there is a risk involved in removing the individual retention from the protocol: could some requirement arise in the future that makes us bring it back to the protocol? One option is to let that field stay for now, and remove it later once we are more certain that it won't be needed back. was (Author: vahid): [~hachikuji] Thank you for your comments. You seem to be looking at this with an inclination to get rid of the retention time from the OffsetCommit protocol. I think with my comments below I'm considering the alternative: # Ewen's KIP proposes to increase the default retention from 1 day to 7 days. So, allowing consumers to set a lower timeout (for the console consumer) seems to be helpful after his KIP; the same way allowing them to set a higher timeout (for actual consumer applications) is helpful before his KIP. # Even if we have offset-level expiration, all offsets in the group should expire together, because the expiration timer starts ticking for all partitions at the same time (when the group becomes empty). The only exception is when a consumer has set a non-default retention time for particular partitions (e.g. using the OffsetCommit API). # Agreed. The expiration timestamp won't make sense. Perhaps the retention time should be stored and whether to expire or not could be calculated on the fly from the time group becomes empty + retention time (we would need to somehow keep the timestamp of the group becoming empty). This expiration check needs to be performed only if the group is empty; otherwise there is no need to expire at all. # I don't have a strong feeling about this. It's for sure simpler to let all offsets expire at the same time. And if we keep the individual offset retention it would be easier to change this in case the cache size becomes an issue. I think there is a risk involved in removing the individual retention from the protocol: could some requirement arise in the future that makes us bring it back to the protocol? One option is to let that field stay for now, and remove it later once we are more certain that it won't be needed back. > Committed offsets should not be deleted if a consumer is still active > - > > Key: KAFKA-4682 > URL: https://issues.apache.org/jira/browse/KAFKA-4682 > Project: Kafka > Issue Type: Bug >Reporter: James Cheng > > Kafka will delete committed offsets that are older than > offsets.retention.minutes > If there is an active consumer on a low traffic partition, it is possible > that Kafka will delete the committed offset for that consumer. Once the > offset is deleted, a restart or a rebalance of that consumer will cause the > consumer to not find any committed offset and start consuming from > earliest/latest (depending on auto.offset.reset). I'm not sure, but a broker > failover might also cause you to start reading from auto.offset.reset (due to > broker restart, or coordinator failover). > I think that Kafka should only delete offsets for inactive consumers. The > timer should only start
[jira] [Commented] (KAFKA-5676) MockStreamsMetrics should be in o.a.k.test
[
https://issues.apache.org/jira/browse/KAFKA-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123846#comment-16123846
]
Guozhang Wang commented on KAFKA-5676:
--
Yeah I think creating a WIP PR and communicate over that diff file would be
faster and more efficient.
> MockStreamsMetrics should be in o.a.k.test
> --
>
> Key: KAFKA-5676
> URL: https://issues.apache.org/jira/browse/KAFKA-5676
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Assignee: Chanchal Singh
> Labels: newbie
> Time Spent: 96h
> Remaining Estimate: 0h
>
> {{MockStreamsMetrics}}'s package should be `o.a.k.test` not
> `o.a.k.streams.processor.internals`.
> In addition, it should not require a {{Metrics}} parameter in its constructor
> as it is only needed for its extended base class; the right way of mocking
> should be implementing {{StreamsMetrics}} with mock behavior than extended a
> real implementaion of {{StreamsMetricsImpl}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5676) MockStreamsMetrics should be in o.a.k.test
[
https://issues.apache.org/jira/browse/KAFKA-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123856#comment-16123856
]
Guozhang Wang commented on KAFKA-5676:
--
I think StreamTaskTest#testMetrics does depend on a real metrics
implementation, hence we should not use MockStreamsMetrics here.
Other classes do not need to access the underlying {{metrics}} value so we
should be able to use MockStreamsMetrics without passing in a metrics object.
> MockStreamsMetrics should be in o.a.k.test
> --
>
> Key: KAFKA-5676
> URL: https://issues.apache.org/jira/browse/KAFKA-5676
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Assignee: Chanchal Singh
> Labels: newbie
> Time Spent: 96h
> Remaining Estimate: 0h
>
> {{MockStreamsMetrics}}'s package should be `o.a.k.test` not
> `o.a.k.streams.processor.internals`.
> In addition, it should not require a {{Metrics}} parameter in its constructor
> as it is only needed for its extended base class; the right way of mocking
> should be implementing {{StreamsMetrics}} with mock behavior than extended a
> real implementaion of {{StreamsMetricsImpl}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-5727) Add the archetype project along with "write applications" web docs.
Guozhang Wang created KAFKA-5727: Summary: Add the archetype project along with "write applications" web docs. Key: KAFKA-5727 URL: https://issues.apache.org/jira/browse/KAFKA-5727 Project: Kafka Issue Type: Sub-task Reporter: Guozhang Wang Assignee: Guozhang Wang -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-5727) Add the archetype project along with "write applications" web docs.
[ https://issues.apache.org/jira/browse/KAFKA-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5727. -- Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request 3630 [https://github.com/apache/kafka/pull/3630] > Add the archetype project along with "write applications" web docs. > --- > > Key: KAFKA-5727 > URL: https://issues.apache.org/jira/browse/KAFKA-5727 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5727) Add the archetype project along with "write applications" web docs.
[ https://issues.apache.org/jira/browse/KAFKA-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123888#comment-16123888 ] ASF GitHub Bot commented on KAFKA-5727: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3630 > Add the archetype project along with "write applications" web docs. > --- > > Key: KAFKA-5727 > URL: https://issues.apache.org/jira/browse/KAFKA-5727 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5727) Add the archetype project along with "write applications" web docs.
[ https://issues.apache.org/jira/browse/KAFKA-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123914#comment-16123914 ] ASF GitHub Bot commented on KAFKA-5727: --- GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/3660 KAFKA-5727: Archetype project for Streams quickstart and tutorial web docs You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka K5727-archetype-project-0110 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3660.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3660 commit b42d0399acef7eb437101c2aef4e6dbd4288f06c Author: Guozhang Wang Date: 2017-08-11T19:19:28Z KAFKA-5727: Add Streams quickstart tutorial as an archetype project 0. Minor fixes on the existing examples to merge all on a single input topic; also do not use `common.utils.Exit` as it is for internal usage only. 1. Add the archetype project for the quickstart. Steps to try it out: a. `mvn install` on the quickstart directory. b. `mvn archetype:generate \ -DarchetypeGroupId=org.apache.kafka \ -DarchetypeArtifactId=streams-quickstart-java \ -DarchetypeVersion=1.0.0-SNAPSHOT \ -DgroupId=streams-quickstart \ -DartifactId=streams-quickstart \ -Dversion=0.1 \ -Dpackage=StreamsQuickstart \ -DinteractiveMode=false` at any directory to create the project. c. build the streams jar with version `1.0.0-SNAPSHOT` to local maven repository with `./gradlew installAll`; `cd streams-quickstart; mvn clean package` d. create the input / output topics, start the console producer and consumer. e. start the program: `mvn exec:java -Dexec.mainClass=StreamsQuickstart.Pipe/LineSplit/WordCount`. f. type data on console producer and observe data on console consumer. Author: Guozhang Wang Reviewers: Damian Guy , Bill Bejeck , Ewen Cheslack-Postava , Eno Thereska Closes #3630 from guozhangwang/KMinor-streams-quickstart-tutorial commit 794b844956f52cbda92352bd4c0befcf909af2f2 Author: Guozhang Wang Date: 2017-08-11T19:39:46Z rebase on 0.11.0 branch > Add the archetype project along with "write applications" web docs. > --- > > Key: KAFKA-5727 > URL: https://issues.apache.org/jira/browse/KAFKA-5727 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5714) Allow whitespaces in the principal name
[
https://issues.apache.org/jira/browse/KAFKA-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123932#comment-16123932
]
Alla Tumarkin commented on KAFKA-5714:
--
Here is the scenario (actual commands below)
- Remove all ACLs
- In server.properties, add the principal as it is in the certificate, without
removing white spaces
{code}
super.users=User:CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
C=Unknown
{code}
- Restart the broker
- Create a topic - will work
- Delete this topic - will mark for deletion but actually will not get deleted
To fix this, repeat the steps from above, with principal name that has no spaces
{code}
super.users=User:CN=Unknown,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown
{code}
Commands
{code}
bin/kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --list
bin/kafka-topics --zookeeper localhost:2181 --create --topic test --partitions
1 --replication-factor 1
bin/kafka-topics --zookeeper localhost:2181 --delete --topic test
{code}
> Allow whitespaces in the principal name
> ---
>
> Key: KAFKA-5714
> URL: https://issues.apache.org/jira/browse/KAFKA-5714
> Project: Kafka
> Issue Type: Improvement
> Components: security
>Affects Versions: 0.10.2.1
>Reporter: Alla Tumarkin
>Assignee: Manikumar
>
> Request
> Improve parser behavior to allow whitespaces in the principal name in the
> config file, as in:
> {code}
> super.users=User:CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown
> {code}
> Background
> Current implementation requires that there are no whitespaces after commas,
> i.e.
> {code}
> super.users=User:CN=Unknown,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown
> {code}
> Note: having a semicolon at the end doesn't help, i.e. this does not work
> either
> {code}
> super.users=User:CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-5728) Stopping consumer thread cause loosing message in the partition
Vasudevan Karnan created KAFKA-5728: --- Summary: Stopping consumer thread cause loosing message in the partition Key: KAFKA-5728 URL: https://issues.apache.org/jira/browse/KAFKA-5728 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.10.1.0 Reporter: Vasudevan Karnan Currently using Spring boot Kafka listener thread to consume the message from partition. Having 10 partitions and concurrency to 10 in the consumer group. In testing, I have 2 messages in the single partition (say for ex: partition 4). Created listener to read the message and post to service. During normal days, read the message and post to service, and working as expected. No issues on that. Suppose if the service is down, then I am doing Spring Retry template to retry to post the message to service (repeatedly) for number of retry and backoff time in ms. If I stop the listener, then getting org.springframework.retry.backoff.BackOffInterruptedException: Thread interrupted while sleeping; nested exception is java.lang.InterruptedException: sleep interrupted at org.springframework.retry.backoff.FixedBackOffPolicy.doBackOff(FixedBackOffPolicy.java:86) ~[spring-retry-1.1.4.RELEASE.jar:na] at org.springframework.retry.backoff.StatelessBackOffPolicy.backOff(StatelessBackOffPolicy.java:36) ~[spring-retry-1.1.4.RELEASE.jar:na] After that I am loosing the message from particular partition (message that are got retried is lost in the middle) and lag is reduced. (This is happening during the end of stopping the listener). Is there any way, we don't loose the message even I am getting the sleep interrupted exception? Suppose during stopping the server, if I dont face sleep interrupt exception, in the next time listener startup, face the same issue and loosing the message again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-4585) KIP-163: Offset fetch and commit requests use the same permissions
[ https://issues.apache.org/jira/browse/KAFKA-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124016#comment-16124016 ] ASF GitHub Bot commented on KAFKA-4585: --- GitHub user vahidhashemian opened a pull request: https://github.com/apache/kafka/pull/3661 KAFKA-4585: Lower the Minimum Required ACL Permission of OffsetFetch (KIP-163) Details can be found in the [KIP](https://cwiki.apache.org/confluence/display/KAFKA/KIP-163%3A+Lower+the+Minimum+Required+ACL+Permission+of+OffsetFetch). You can merge this pull request into a Git repository by running: $ git pull https://github.com/vahidhashemian/kafka KAFKA-4585 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3661.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3661 commit be9e43db15952122836618520b7a166641c9914d Author: Vahid Hashemian Date: 2017-08-11T20:32:56Z KAFKA-4585: Lower the Minimum Required ACL Permission of OffsetFetch (KIP-163) Details can be found in the [KIP](https://cwiki.apache.org/confluence/display/KAFKA/KIP-163%3A+Lower+the+Minimum+Required+ACL+Permission+of+OffsetFetch). > KIP-163: Offset fetch and commit requests use the same permissions > -- > > Key: KAFKA-4585 > URL: https://issues.apache.org/jira/browse/KAFKA-4585 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.10.1.1 >Reporter: Ewen Cheslack-Postava >Assignee: Vahid Hashemian > Labels: kip > > Currently the handling of permissions for consumer groups seems a bit odd > because most of the requests use the Read permission on the Group (join, > sync, heartbeat, leave, offset commit, and offset fetch). This means you > cannot lock down certain functionality for certain users. For this issue I'll > highlight a realistic issue since conflating the ability to perform most of > these operations may not be a serious issue. > In particular, if you want tooling for monitoring offsets (i.e. you want to > be able to read from all groups) but don't want that tool to be able to write > offsets, you currently cannot achieve this. Part of the reason this seems odd > to me is that any operation which can mutate state seems like it should be a > Write operation (i.e. joining, syncing, leaving, and committing; maybe > heartbeat as well). However, [~hachikuji] has mentioned that the use of Read > may have been intentional. If that is the case, changing at least offset > fetch to be a Describe operation instead would allow isolating the mutating > vs non-mutating request types. > Note that this would require a KIP and would potentially have some > compatibility implications. Note however, that if we went with the Describe > option, Describe is allowed by default when Read, Write, or Delete are > allowed, so this may not have to have any compatibility issues (if the user > previously allowed Read, they'd still have all the same capabilities as > before). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-1379) Partition reassignment resets clock for time-based retention
[ https://issues.apache.org/jira/browse/KAFKA-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson resolved KAFKA-1379. - Resolution: Fixed Assignee: Jiangjie Qin Fix Version/s: 0.10.1.0 Marking this bug as resolved in 0.10.1.0 based on the statement "The log retention time is no longer based on last modified time of the log segments. Instead it will be based on the largest timestamp of the messages in a log segment." in http://kafka.apache.org/documentation.html#upgrade_10_1_breaking > Partition reassignment resets clock for time-based retention > > > Key: KAFKA-1379 > URL: https://issues.apache.org/jira/browse/KAFKA-1379 > Project: Kafka > Issue Type: Bug > Components: log >Reporter: Joel Koshy >Assignee: Jiangjie Qin > Fix For: 0.10.1.0 > > > Since retention is driven off mod-times reassigned partitions will result in > data that has been on a leader to be retained for another full retention > cycle. E.g., if retention is seven days and you reassign partitions on the > sixth day then those partitions will remain on the replicas for another > seven days. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5138) MirrorMaker doesn't exit on send failure occasionally
[
https://issues.apache.org/jira/browse/KAFKA-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124106#comment-16124106
]
Dustin Cote commented on KAFKA-5138:
[~tamas.mate] I've never managed to reproduce it. It came out of a poorly
performing destination cluster, so instead of iptables, you could try running
with an extremely low `request.timeout.ms` on the mirrormaker producer?
> MirrorMaker doesn't exit on send failure occasionally
> -
>
> Key: KAFKA-5138
> URL: https://issues.apache.org/jira/browse/KAFKA-5138
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.10.2.0
>Reporter: Dustin Cote
> Labels: newbie
>
> MirrorMaker with abort.on.send.failure=true does not always exit if the
> producer closes. Here is the logic that happens:
> First we encounter a problem producing and force the producer to close
> {code}
> [2017-04-10 07:17:25,137] ERROR Error when sending message to topic
> mytopicwith key: 20 bytes, value: 314 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 47 record(s) for
> mytopic-2: 30879 ms has passed since last append
> [2017-04-10 07:17:25,170] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2017-04-10 07:17:25,170] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:25,170] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:25,170] DEBUG The Kafka producer has closed.
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:25,170] ERROR Error when sending message to topic mytopic
> with key: 20 bytes, value: 313 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 47 record(s) for
> mytopic-2: 30879 ms has passed since last append
> [2017-04-10 07:17:25,170] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2017-04-10 07:17:25,170] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:25,170] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:25,170] DEBUG The Kafka producer has closed.
> (org.apache.kafka.clients.producer.KafkaProducer)
> {code}
> All good there. Then we can't seem to close the producer nicely after about
> 15 seconds and so it is forcefully killed:
> {code}
> [2017-04-10 07:17:39,778] ERROR Error when sending message to topic
> mytopic.subscriptions with key: 70 bytes, value: null with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> java.lang.IllegalStateException: Producer is closed forcefully.
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:522)
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:502)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:147)
> at java.lang.Thread.run(Unknown Source)
> [2017-04-10 07:17:39,778] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2017-04-10 07:17:39,778] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:39,778] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:39,778] DEBUG The Kafka producer has closed.
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:39,779] DEBUG Removed sensor with name connections-closed:
> (org.apache.kafka.common.metrics.Metrics)
> {code}
> After removing some metric sensors for awhile this happens:
> {code}
> [2017-04-10 07:17:39,780] DEBUG Removed sensor with name node-3.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2017-04-10 07:17:39,780] DEBUG Shutdown of Kafka producer I/O thread has
> completed. (org.apache.kafka.clients.producer.internals.Sender)
> [2017-04-10 07:17:41,852] DEBUG Sending Heartbeat request for group
> mirror-maker-1491619052-teab1-1 to coordinator myhost1:9092 (id: 2147483643
> rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2017-04-10 07:17:41,953] DEBUG Received successful Heartbeat response for
> group mirror-maker-1491619052-teab1-1
>
[jira] [Commented] (KAFKA-5138) MirrorMaker doesn't exit on send failure occasionally
[
https://issues.apache.org/jira/browse/KAFKA-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124149#comment-16124149
]
Tamas Mate commented on KAFKA-5138:
---
[~cote] Cool, thanks for the tip, I have just installed Jmeter, then I will
keep going this way. :)
> MirrorMaker doesn't exit on send failure occasionally
> -
>
> Key: KAFKA-5138
> URL: https://issues.apache.org/jira/browse/KAFKA-5138
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.10.2.0
>Reporter: Dustin Cote
> Labels: newbie
>
> MirrorMaker with abort.on.send.failure=true does not always exit if the
> producer closes. Here is the logic that happens:
> First we encounter a problem producing and force the producer to close
> {code}
> [2017-04-10 07:17:25,137] ERROR Error when sending message to topic
> mytopicwith key: 20 bytes, value: 314 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 47 record(s) for
> mytopic-2: 30879 ms has passed since last append
> [2017-04-10 07:17:25,170] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2017-04-10 07:17:25,170] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:25,170] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:25,170] DEBUG The Kafka producer has closed.
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:25,170] ERROR Error when sending message to topic mytopic
> with key: 20 bytes, value: 313 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 47 record(s) for
> mytopic-2: 30879 ms has passed since last append
> [2017-04-10 07:17:25,170] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2017-04-10 07:17:25,170] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:25,170] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:25,170] DEBUG The Kafka producer has closed.
> (org.apache.kafka.clients.producer.KafkaProducer)
> {code}
> All good there. Then we can't seem to close the producer nicely after about
> 15 seconds and so it is forcefully killed:
> {code}
> [2017-04-10 07:17:39,778] ERROR Error when sending message to topic
> mytopic.subscriptions with key: 70 bytes, value: null with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> java.lang.IllegalStateException: Producer is closed forcefully.
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:522)
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:502)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:147)
> at java.lang.Thread.run(Unknown Source)
> [2017-04-10 07:17:39,778] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2017-04-10 07:17:39,778] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:39,778] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:39,778] DEBUG The Kafka producer has closed.
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-04-10 07:17:39,779] DEBUG Removed sensor with name connections-closed:
> (org.apache.kafka.common.metrics.Metrics)
> {code}
> After removing some metric sensors for awhile this happens:
> {code}
> [2017-04-10 07:17:39,780] DEBUG Removed sensor with name node-3.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2017-04-10 07:17:39,780] DEBUG Shutdown of Kafka producer I/O thread has
> completed. (org.apache.kafka.clients.producer.internals.Sender)
> [2017-04-10 07:17:41,852] DEBUG Sending Heartbeat request for group
> mirror-maker-1491619052-teab1-1 to coordinator myhost1:9092 (id: 2147483643
> rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2017-04-10 07:17:41,953] DEBUG Received successful Heartbeat response for
> group mirror-maker-1491619052-teab1-1
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2017-04-10 07:17:44,875] DEBUG Sending Heartbeat request f
[jira] [Created] (KAFKA-5729) Consumer should verify offset commits are from assigned partitions
Jason Gustafson created KAFKA-5729: -- Summary: Consumer should verify offset commits are from assigned partitions Key: KAFKA-5729 URL: https://issues.apache.org/jira/browse/KAFKA-5729 Project: Kafka Issue Type: Bug Components: consumer Reporter: Jason Gustafson Fix For: 1.0.0 Need to think through the compatibility implications since we currently allow this, but at a minimum, we should verify that only offsets from partitions dynamically assigned can be committed. The lack of this validation tends to mask problems in the partition revocation and assignment process. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5730) Consumer should ensure consistent ordering between sync and async offset commits
Jason Gustafson created KAFKA-5730:
--
Summary: Consumer should ensure consistent ordering between sync
and async offset commits
Key: KAFKA-5730
URL: https://issues.apache.org/jira/browse/KAFKA-5730
Project: Kafka
Issue Type: Bug
Components: consumer
Reporter: Jason Gustafson
Assignee: Jason Gustafson
Fix For: 1.0.0
If {{commitSync}} is called with an async offset commit in-flight, it is
possible (even likely) for it to return prior to the invocation of the async
callback even though the OffsetCommit request for the latter will have actually
completed first. In this case, the async callback is not invoked until the next
call to {{poll}}, which can lead to very confusing behavior since the client is
led to think that the async commit happened second.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-5731) Connect WorkerSinkTask out of order offset commit can lead to inconsistent state
Jason Gustafson created KAFKA-5731:
--
Summary: Connect WorkerSinkTask out of order offset commit can
lead to inconsistent state
Key: KAFKA-5731
URL: https://issues.apache.org/jira/browse/KAFKA-5731
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Jason Gustafson
Fix For: 1.0.0
In Connect's WorkerSinkTask, we do sequence number validation to ensure that
offset commits are handled in the right order
(https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L199).
Unfortunately, for asynchronous commits, the {{lastCommittedOffsets}} field is
overridden regardless of this sequence check as long as the response had no
error
(https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L284).
Hence if we get an out of order commit, then the internal state will be
inconsistent. To fix this, we should only override {{lastCommittedOffsets}}
after sequence validation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (KAFKA-5731) Connect WorkerSinkTask out of order offset commit can lead to inconsistent state
[
https://issues.apache.org/jira/browse/KAFKA-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch reassigned KAFKA-5731:
Assignee: Randall Hauch
> Connect WorkerSinkTask out of order offset commit can lead to inconsistent
> state
>
>
> Key: KAFKA-5731
> URL: https://issues.apache.org/jira/browse/KAFKA-5731
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Jason Gustafson
>Assignee: Randall Hauch
> Fix For: 1.0.0
>
>
> In Connect's WorkerSinkTask, we do sequence number validation to ensure that
> offset commits are handled in the right order
> (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L199).
>
> Unfortunately, for asynchronous commits, the {{lastCommittedOffsets}} field
> is overridden regardless of this sequence check as long as the response had
> no error
> (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L284).
>
> Hence if we get an out of order commit, then the internal state will be
> inconsistent. To fix this, we should only override {{lastCommittedOffsets}}
> after sequence validation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5074) Transition to OnlinePartition without preferred leader in ISR fails
[
https://issues.apache.org/jira/browse/KAFKA-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124325#comment-16124325
]
Pierre Mage commented on KAFKA-5074:
Running 0.11.0 and observing similar behaviour.
Sequence of events recorded in logs:
1. ZooKeeper session expires
2. Kafka controller stops broker 0
3. Kafka re-register broker 0 in ZooKeeper
4. Leader cache \[mytopic,29\] ->
(Leader:2,ISR:2,0,LeaderEpoch:0,ControllerEpoch:1)
5. Invoking state change to OfflineReplica for replicas
\[Topic=mytopic,Partition=29,Replica=0\]
6. Retaining last ISR 0 of partition \[mytopic,29\] since unclean leader
election is disabled
7. New leader and ISR for partition \[mytopic,29\] is
{"leader":-1,"leader_epoch":4,"isr":[0]}
8. Not sending request (type=StopReplicaRequest...) to broker 0, since it is
offline
9. Invoking state change to OnlineReplica for replicas
\[Topic=mytopic,Partition=29,Replica=0\]
10. Cycle of failing preferred leader elections starts
OfflinePartitionLeaderSelector is not called as the partition's state is still
OnlinePartition.
{code}
ERROR Controller 2 epoch 4 encountered error while electing leader for
partition [mytopic,29] due to: Preferred replica 2 for partition [mytopci,29]
is either not alive or not in the isr. Current leader and ISR
[{"leader":-1,"leader_epoch":4,"isr":[0]}].
ERROR Controller 2 epoch 4 initiated state change for partition [mytopic,29]
from OnlinePartition to OnlinePartition failed
{code}
> Transition to OnlinePartition without preferred leader in ISR fails
> ---
>
> Key: KAFKA-5074
> URL: https://issues.apache.org/jira/browse/KAFKA-5074
> Project: Kafka
> Issue Type: Bug
> Components: controller
>Affects Versions: 0.9.0.0
>Reporter: Dustin Cote
>
> Running 0.9.0.0, the controller can get into a state where it no longer is
> able to elect a leader for an Offline partition. It's unclear how this state
> is first achieved but in the steady state, this happens:
> -There are partitions with a leader of -1
> -The Controller repeatedly attempts a preferred leader election for these
> partitions
> -The preferred leader election fails because the only replica in the ISR is
> not the preferred leader
> The log cycle looks like this:
> {code}
> [2017-04-12 18:00:18,891] INFO [Controller 8]: Starting preferred replica
> leader election for partitions topic,1
> [2017-04-12 18:00:18,891] INFO [Partition state machine on Controller 8]:
> Invoking state change to OnlinePartition for partitions topic,1
> [2017-04-12 18:00:18,892] INFO [PreferredReplicaPartitionLeaderSelector]:
> Current leader -1 for partition [topic,1] is not the preferred replica.
> Trigerring preferred replica leader election
> (kafka.controller.PreferredReplicaPartitionLeaderSelector)
> [2017-04-12 18:00:18,893] WARN [Controller 8]: Partition [topic,1] failed to
> complete preferred replica leader election. Leader is -1
> (kafka.controller.KafkaController)
> {code}
> It's not clear if this would happen on versions later that 0.9.0.0.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
