[jira] [Resolved] (KAFKA-5700) producer missed header information when splitting batches
[ https://issues.apache.org/jira/browse/KAFKA-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huxihx resolved KAFKA-5700. --- Resolution: Fixed > producer missed header information when splitting batches > - > > Key: KAFKA-5700 > URL: https://issues.apache.org/jira/browse/KAFKA-5700 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 0.11.0.0 >Reporter: huxihx >Assignee: huxihx >Priority: Critical > Labels: reliability > Fix For: 0.11.0.1 > > > In `ProducerBatch.tryAppendForSplit`, invoking > `this.recordsBuilder.append(timestamp, key, value);` missed the header > information in the ProducerRecord. Should invoke this like : > `this.recordsBuilder.append(timestamp, key, value, headers);` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5700) producer missed header information when splitting batches
[ https://issues.apache.org/jira/browse/KAFKA-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116066#comment-16116066 ] ASF GitHub Bot commented on KAFKA-5700: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3620 > producer missed header information when splitting batches > - > > Key: KAFKA-5700 > URL: https://issues.apache.org/jira/browse/KAFKA-5700 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 0.11.0.0 >Reporter: huxihx >Assignee: huxihx >Priority: Critical > Labels: reliability > Fix For: 0.11.0.1 > > > In `ProducerBatch.tryAppendForSplit`, invoking > `this.recordsBuilder.append(timestamp, key, value);` missed the header > information in the ProducerRecord. Should invoke this like : > `this.recordsBuilder.append(timestamp, key, value, headers);` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5643) Using _DUCKTAPE_OPTIONS has no effect on executing tests
[ https://issues.apache.org/jira/browse/KAFKA-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116051#comment-16116051 ] ASF GitHub Bot commented on KAFKA-5643: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3578 > Using _DUCKTAPE_OPTIONS has no effect on executing tests > > > Key: KAFKA-5643 > URL: https://issues.apache.org/jira/browse/KAFKA-5643 > Project: Kafka > Issue Type: Bug > Components: system tests >Reporter: Paolo Patierno >Assignee: Paolo Patierno > Fix For: 0.10.2.2, 0.11.0.1, 1.0.0 > > > Hi, > as described in the documentation, you should be able to enable debugging > using the following line : > _DUCKTAPE_OPTIONS="--debug" bash tests/docker/run_tests.sh | tee > debug_logs.txt > Instead the _DUCKTAPE_OPTIONS isn't available in the run_tests.sh script so > it's not passed to the ducker-ak and finally on the ducktape command line. > Thanks, > Paolo. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5643) Using _DUCKTAPE_OPTIONS has no effect on executing tests
[ https://issues.apache.org/jira/browse/KAFKA-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-5643: - Fix Version/s: 0.11.0.1 > Using _DUCKTAPE_OPTIONS has no effect on executing tests > > > Key: KAFKA-5643 > URL: https://issues.apache.org/jira/browse/KAFKA-5643 > Project: Kafka > Issue Type: Bug > Components: system tests >Reporter: Paolo Patierno >Assignee: Paolo Patierno > Fix For: 0.10.2.2, 0.11.0.1, 1.0.0 > > > Hi, > as described in the documentation, you should be able to enable debugging > using the following line : > _DUCKTAPE_OPTIONS="--debug" bash tests/docker/run_tests.sh | tee > debug_logs.txt > Instead the _DUCKTAPE_OPTIONS isn't available in the run_tests.sh script so > it's not passed to the ducker-ak and finally on the ducktape command line. > Thanks, > Paolo. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5643) Using _DUCKTAPE_OPTIONS has no effect on executing tests
[ https://issues.apache.org/jira/browse/KAFKA-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116046#comment-16116046 ] Ewen Cheslack-Postava commented on KAFKA-5643: -- [~ppatierno] We don't really consider any changes in test code that isn't meant for public use to be subject to regressions, but it's fine to add this back in if it affected anyone's use of the script. > Using _DUCKTAPE_OPTIONS has no effect on executing tests > > > Key: KAFKA-5643 > URL: https://issues.apache.org/jira/browse/KAFKA-5643 > Project: Kafka > Issue Type: Bug > Components: system tests >Reporter: Paolo Patierno >Assignee: Paolo Patierno > > Hi, > as described in the documentation, you should be able to enable debugging > using the following line : > _DUCKTAPE_OPTIONS="--debug" bash tests/docker/run_tests.sh | tee > debug_logs.txt > Instead the _DUCKTAPE_OPTIONS isn't available in the run_tests.sh script so > it's not passed to the ducker-ak and finally on the ducktape command line. > Thanks, > Paolo. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5707) Remove useless `--force` option for both TopicCommand and ConfigCommand
[ https://issues.apache.org/jira/browse/KAFKA-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116001#comment-16116001 ] ASF GitHub Bot commented on KAFKA-5707: --- GitHub user huxihx opened a pull request: https://github.com/apache/kafka/pull/3632 KAFKA-5707: TopicCommand and ConfigCommand should remove useless `--force` option. TopicCommand and ConfigCommand should remove useless `--force` option. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huxihx/kafka KAFKA-5707 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3632.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3632 commit d16d9882c1fe0daf2ce2809395020f2289f13f77 Author: huxihx Date: 2017-08-07T01:40:36Z KAFKA-5707: TopicCommand and ConfigCommand should remove useless `--force` option. > Remove useless `--force` option for both TopicCommand and ConfigCommand > --- > > Key: KAFKA-5707 > URL: https://issues.apache.org/jira/browse/KAFKA-5707 > Project: Kafka > Issue Type: Bug > Components: admin >Affects Versions: 0.11.0.0 >Reporter: huxihx >Assignee: huxihx >Priority: Minor > > `TopicCommand` and `ConfigCommand` do expose an option named `--force` which > suppresses console prompts, but both classes do not actually use it. Should > remove it from the usage description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5707) Remove useless `--force` option for both TopicCommand and ConfigCommand
huxihx created KAFKA-5707: - Summary: Remove useless `--force` option for both TopicCommand and ConfigCommand Key: KAFKA-5707 URL: https://issues.apache.org/jira/browse/KAFKA-5707 Project: Kafka Issue Type: Bug Components: admin Affects Versions: 0.11.0.0 Reporter: huxihx Assignee: huxihx Priority: Minor `TopicCommand` and `ConfigCommand` do expose an option named `--force` which suppresses console prompts, but both classes do not actually use it. Should remove it from the usage description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5676) MockStreamsMetrics should be in o.a.k.test
[
https://issues.apache.org/jira/browse/KAFKA-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115860#comment-16115860
]
Chanchal Singh edited comment on KAFKA-5676 at 8/6/17 5:32 PM:
---
Thanks Wang . I tried doing that but found that in few test cases they are
dependent on Metrics object modified by StreamsMetricsImpl object and the value
of Metrics object is then tested. I am confused about ,how MockStreamMetrics
is acting as mock class if its not returning any mocking behaviour.
Is there any difference in using StreamsMetricsImpl directly instead of
MockStreamMetrics class in current case?
was (Author: chanchal.kafka):
Thanks Wang . I tried doing that but found that in few test cases they are
dependent on Metrics object modified by StreamsMetricsImpl object and the value
of Metrics object is then tested. I am also confused how MockStreamMetrics is
acting as mock class ff its not returning any mocking behaviour.
Is there any difference in using StreamsMetricsImpl directly instead of
MockStreamMetrics class in current case?
> MockStreamsMetrics should be in o.a.k.test
> --
>
> Key: KAFKA-5676
> URL: https://issues.apache.org/jira/browse/KAFKA-5676
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Assignee: Chanchal Singh
> Labels: newbie
>
> {{MockStreamsMetrics}}'s package should be `o.a.k.test` not
> `o.a.k.streams.processor.internals`.
> In addition, it should not require a {{Metrics}} parameter in its constructor
> as it is only needed for its extended base class; the right way of mocking
> should be implementing {{StreamsMetrics}} with mock behavior than extended a
> real implementaion of {{StreamsMetricsImpl}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5676) MockStreamsMetrics should be in o.a.k.test
[
https://issues.apache.org/jira/browse/KAFKA-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115860#comment-16115860
]
Chanchal Singh edited comment on KAFKA-5676 at 8/6/17 5:27 PM:
---
Thanks Wang . I tried doing that but found that in few test cases they are
dependent on Metrics object modified by StreamsMetricsImpl object and the value
of Metrics object is then tested. I am also confused how MockStreamMetrics is
acting as mock class ff its not returning any mocking behaviour.
Is there any difference in using StreamsMetricsImpl directly instead of
MockStreamMetrics class in current case?
was (Author: chanchal.kafka):
Thanks Wang . I tried doing that but found that in few test cases they are
dependent on Metrics object modified by StreamsMetricsImpl object and the value
of Metrics object is then tested. I am also confused how MockStreamMetrics is
acting as mock class ff its not returning any mocking behaviour.
Is there any difference in using StreamsMetricsImpl directly instead of
MockStreamMetrics class ?
> MockStreamsMetrics should be in o.a.k.test
> --
>
> Key: KAFKA-5676
> URL: https://issues.apache.org/jira/browse/KAFKA-5676
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Assignee: Chanchal Singh
> Labels: newbie
>
> {{MockStreamsMetrics}}'s package should be `o.a.k.test` not
> `o.a.k.streams.processor.internals`.
> In addition, it should not require a {{Metrics}} parameter in its constructor
> as it is only needed for its extended base class; the right way of mocking
> should be implementing {{StreamsMetrics}} with mock behavior than extended a
> real implementaion of {{StreamsMetricsImpl}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5676) MockStreamsMetrics should be in o.a.k.test
[
https://issues.apache.org/jira/browse/KAFKA-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115860#comment-16115860
]
Chanchal Singh commented on KAFKA-5676:
---
Thanks Wang . I tried doing that but found that in few test cases they are
dependent on Metrics object modified by StreamsMetricsImpl object and the value
of Metrics object is then tested. I am also confused how MockStreamMetrics is
acting as mock class ff its not returning any mocking behaviour.
Is there any difference in using StreamsMetricsImpl directly instead of
MockStreamMetrics class ?
> MockStreamsMetrics should be in o.a.k.test
> --
>
> Key: KAFKA-5676
> URL: https://issues.apache.org/jira/browse/KAFKA-5676
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Assignee: Chanchal Singh
> Labels: newbie
>
> {{MockStreamsMetrics}}'s package should be `o.a.k.test` not
> `o.a.k.streams.processor.internals`.
> In addition, it should not require a {{Metrics}} parameter in its constructor
> as it is only needed for its extended base class; the right way of mocking
> should be implementing {{StreamsMetrics}} with mock behavior than extended a
> real implementaion of {{StreamsMetricsImpl}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5546) Temporary loss of availability data when the leader is disconnected
[
https://issues.apache.org/jira/browse/KAFKA-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115735#comment-16115735
]
Björn Eriksson commented on KAFKA-5546:
---
Hi Jason,
No, {{ifdown}} means that the connection won't be shut down cleanly. We're
building a fault tolerant system and we need to test network failures, like
hardware failure or a disconnected network cable.
I've updated the 0.11.0.0 branch to include results for {{ifdown}} and {{kill
-9}} (docker rm). Testing with {{kill -9}} shows better results (2 - 8 seconds)
but we'd like guarantees much lower than that.
The {{ifdown}} test shows that after the _1003_ leader is disconnected
(_@11:31:12_) it takes ~2.5 seconds for the producer to realise this and report
_Disconnecting from node 1003 due to request timeout_. Zookeeper reports the
new leader to be _1002_ after ~ 6 seconds but the producer doesn't get wind of
the new leader until 14 seconds after the network failure in spite of it
continuously sending metadata requests.
> Temporary loss of availability data when the leader is disconnected
> ---
>
> Key: KAFKA-5546
> URL: https://issues.apache.org/jira/browse/KAFKA-5546
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 0.10.2.1, 0.11.0.0
> Environment: docker, failing-network
>Reporter: Björn Eriksson
>
> We've noticed that if the leaders networking is deconfigured (with {{ifconfig
> eth0 down}}) the producer won't notice this and doesn't immediately connect
> to the newly elected leader.
> {{docker-compose.yml}} and test runner are at
> https://github.com/owbear/kafka-network-failure-tests.
> We were expecting a transparent failover to the new leader but testing shows
> that there's a 8-15 seconds long gap where no values are stored in the log
> after the network is taken down.
> Tests (and results) [against
> 0.10.2.1|https://github.com/owbear/kafka-network-failure-tests/tree/kafka-network-failure-tests-0.10.2.1]
> Tests (and results) [against
> 0.11.0.0|https://github.com/owbear/kafka-network-failure-tests/tree/kafka-network-failure-tests-0.11.0.0]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5430) new consumers getting data for revoked partitions
[
https://issues.apache.org/jira/browse/KAFKA-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115732#comment-16115732
]
Lior Chaga edited comment on KAFKA-5430 at 8/6/17 9:49 AM:
---
Issue reproduced again, log of thread running consumer attached
[^consumer-thread.log].
You can see:
{code}
2017-08-06 05:23:43,405 INFO [Kafka Topics Consumer
sessionlegacy.consumer-17_session_parser_01] ConsumerCoordinator [] - Revoking
previously assigned partitions [sessionlegacy-5, sessionlegacy-4,
sessionlegacy-3] for group session_parser_01
{code}
But no "Successfully joined group {} with generation {}" message appear after
revoking, and no "Setting newly assigned partitions".
Seems to me the consumer is stuck in MemberState.REBALANCING, yet still getting
data.
was (Author: liorchaga):
Issue reproduced again, log of thread running consumer attached.
You can see:
{code}
2017-08-06 05:23:43,405 INFO [Kafka Topics Consumer
sessionlegacy.consumer-17_session_parser_01] ConsumerCoordinator [] - Revoking
previously assigned partitions [sessionlegacy-5, sessionlegacy-4,
sessionlegacy-3] for group session_parser_01
{code}
But no "Successfully joined group {} with generation {}" message appear after
revoking, and no "Setting newly assigned partitions".
Seems to me the consumer is stuck in MemberState.REBALANCING, yet still getting
data.
> new consumers getting data for revoked partitions
> -
>
> Key: KAFKA-5430
> URL: https://issues.apache.org/jira/browse/KAFKA-5430
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 0.10.2.0
>Reporter: Lior Chaga
> Attachments: consumer-thread.log, consumer-thread.log,
> kafka_trace.log.gz
>
>
> Due to bad configuration applied to network components, we experienced issues
> with communication between kafka brokers (causing under replication) as well
> as producers/consumers not being able to work against kafka.
> The symptoms on the consumer were many errors of the following form:
> {code}
> 2017-06-04 04:27:35,200 ERROR [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] TaboolaKafkaConsumer [] - Failed
> committing to kafka topicPartitions
> [requestlegacy-2,requestlegacy-0,requestlegacy-1]
> org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset
> commit failed with a retriable exception. You should retry committing offsets.
> Caused by: org.apache.kafka.common.errors.DisconnectException
> {code}
> So far so good. However, upon network recovery, there were several rebalance
> operations, which eventually resulted in only one consumer (#14) being
> assigned with all topic partitions (at this case we're talking about a
> consumer groups for which all consumers are running in same process):
> {code}
> 2017-06-04 04:27:02,168 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-14_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-8, requestlegacy-9]
> for group session_parser_02
> 2017-06-04 04:27:04,208 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-15_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-10, requestlegacy-11]
> for group session_parser_02
> 2017-06-04 04:27:18,167 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-12_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-3, requestlegacy-4,
> requestlegacy-5] for group session_parser_02
> 2017-06-04 04:27:20,232 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-2, requestlegacy-0,
> requestlegacy-1] for group session_parser_02
> 2017-06-04 04:27:20,236 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-15_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-9, requestlegacy-10,
> requestlegacy-11] for group session_parser_02
> 2017-06-04 04:27:20,237 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-12_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-3, requestlegacy-4, requestlegacy-5]
> for group session_parser_02
> 2017-06-04 04:27:20,237 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-14_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-6, requestlegacy-7, requestlegacy-8]
> for group session_parser_02
> 2017-06-04 04:27:20,332 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-2, requestlegacy-0, requestlegacy-1]
> for group session_parser_02
> 2017-06-04 04:28:52
[jira] [Updated] (KAFKA-5430) new consumers getting data for revoked partitions
[
https://issues.apache.org/jira/browse/KAFKA-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Chaga updated KAFKA-5430:
--
Attachment: consumer-thread.log
> new consumers getting data for revoked partitions
> -
>
> Key: KAFKA-5430
> URL: https://issues.apache.org/jira/browse/KAFKA-5430
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 0.10.2.0
>Reporter: Lior Chaga
> Attachments: consumer-thread.log, consumer-thread.log,
> kafka_trace.log.gz
>
>
> Due to bad configuration applied to network components, we experienced issues
> with communication between kafka brokers (causing under replication) as well
> as producers/consumers not being able to work against kafka.
> The symptoms on the consumer were many errors of the following form:
> {code}
> 2017-06-04 04:27:35,200 ERROR [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] TaboolaKafkaConsumer [] - Failed
> committing to kafka topicPartitions
> [requestlegacy-2,requestlegacy-0,requestlegacy-1]
> org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset
> commit failed with a retriable exception. You should retry committing offsets.
> Caused by: org.apache.kafka.common.errors.DisconnectException
> {code}
> So far so good. However, upon network recovery, there were several rebalance
> operations, which eventually resulted in only one consumer (#14) being
> assigned with all topic partitions (at this case we're talking about a
> consumer groups for which all consumers are running in same process):
> {code}
> 2017-06-04 04:27:02,168 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-14_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-8, requestlegacy-9]
> for group session_parser_02
> 2017-06-04 04:27:04,208 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-15_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-10, requestlegacy-11]
> for group session_parser_02
> 2017-06-04 04:27:18,167 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-12_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-3, requestlegacy-4,
> requestlegacy-5] for group session_parser_02
> 2017-06-04 04:27:20,232 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-2, requestlegacy-0,
> requestlegacy-1] for group session_parser_02
> 2017-06-04 04:27:20,236 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-15_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-9, requestlegacy-10,
> requestlegacy-11] for group session_parser_02
> 2017-06-04 04:27:20,237 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-12_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-3, requestlegacy-4, requestlegacy-5]
> for group session_parser_02
> 2017-06-04 04:27:20,237 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-14_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-6, requestlegacy-7, requestlegacy-8]
> for group session_parser_02
> 2017-06-04 04:27:20,332 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-2, requestlegacy-0, requestlegacy-1]
> for group session_parser_02
> 2017-06-04 04:28:52,368 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-13_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-6, requestlegacy-7]
> for group session_parser_02
> 2017-06-04 04:29:15,201 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-2, requestlegacy-0,
> requestlegacy-1] for group session_parser_02
> 2017-06-04 04:30:22,379 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-14_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-6, requestlegacy-7,
> requestlegacy-8] for group session_parser_02
> 2017-06-04 04:30:24,431 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-15_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-9, requestlegacy-10,
> requestlegacy-11] for group session_parser_02
> 2017-06-04 04:30:38,229 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-12_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-3, requestlegacy-4,
> requestlegacy-5] for group session_parser_02
> 2017-06-04 04
[jira] [Updated] (KAFKA-5430) new consumers getting data for revoked partitions
[
https://issues.apache.org/jira/browse/KAFKA-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Chaga updated KAFKA-5430:
--
Attachment: consumer-thread.log
Issue reproduced again, log of thread running consumer attached.
You can see:
{code}
2017-08-06 05:23:43,405 INFO [Kafka Topics Consumer
sessionlegacy.consumer-17_session_parser_01] ConsumerCoordinator [] - Revoking
previously assigned partitions [sessionlegacy-5, sessionlegacy-4,
sessionlegacy-3] for group session_parser_01
{code}
But no "Successfully joined group {} with generation {}" message appear after
revoking, and no "Setting newly assigned partitions".
Seems to me the consumer is stuck in MemberState.REBALANCING, yet still getting
data.
> new consumers getting data for revoked partitions
> -
>
> Key: KAFKA-5430
> URL: https://issues.apache.org/jira/browse/KAFKA-5430
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 0.10.2.0
>Reporter: Lior Chaga
> Attachments: consumer-thread.log, kafka_trace.log.gz
>
>
> Due to bad configuration applied to network components, we experienced issues
> with communication between kafka brokers (causing under replication) as well
> as producers/consumers not being able to work against kafka.
> The symptoms on the consumer were many errors of the following form:
> {code}
> 2017-06-04 04:27:35,200 ERROR [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] TaboolaKafkaConsumer [] - Failed
> committing to kafka topicPartitions
> [requestlegacy-2,requestlegacy-0,requestlegacy-1]
> org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset
> commit failed with a retriable exception. You should retry committing offsets.
> Caused by: org.apache.kafka.common.errors.DisconnectException
> {code}
> So far so good. However, upon network recovery, there were several rebalance
> operations, which eventually resulted in only one consumer (#14) being
> assigned with all topic partitions (at this case we're talking about a
> consumer groups for which all consumers are running in same process):
> {code}
> 2017-06-04 04:27:02,168 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-14_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-8, requestlegacy-9]
> for group session_parser_02
> 2017-06-04 04:27:04,208 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-15_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-10, requestlegacy-11]
> for group session_parser_02
> 2017-06-04 04:27:18,167 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-12_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-3, requestlegacy-4,
> requestlegacy-5] for group session_parser_02
> 2017-06-04 04:27:20,232 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-2, requestlegacy-0,
> requestlegacy-1] for group session_parser_02
> 2017-06-04 04:27:20,236 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-15_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-9, requestlegacy-10,
> requestlegacy-11] for group session_parser_02
> 2017-06-04 04:27:20,237 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-12_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-3, requestlegacy-4, requestlegacy-5]
> for group session_parser_02
> 2017-06-04 04:27:20,237 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-14_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-6, requestlegacy-7, requestlegacy-8]
> for group session_parser_02
> 2017-06-04 04:27:20,332 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] ConsumerCoordinator [] - Setting
> newly assigned partitions [requestlegacy-2, requestlegacy-0, requestlegacy-1]
> for group session_parser_02
> 2017-06-04 04:28:52,368 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-13_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-6, requestlegacy-7]
> for group session_parser_02
> 2017-06-04 04:29:15,201 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-11_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-2, requestlegacy-0,
> requestlegacy-1] for group session_parser_02
> 2017-06-04 04:30:22,379 INFO [Kafka Topics Cosumer
> requestlegacy.consumer-14_session_parser_02] ConsumerCoordinator [] -
> Revoking previously assigned partitions [requestlegacy-6, requestlegacy-7,
> requestlegacy-8] for group se
