[jira] [Comment Edited] (KAFKA-7026) Sticky assignor could assign a partition to multiple consumers (KIP-341)
[
https://issues.apache.org/jira/browse/KAFKA-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932965#comment-16932965
]
redbrick9 edited comment on KAFKA-7026 at 9/19/19 1:20 AM:
---
We are testing our application based on kafka, when we are using iptables to
drop/reject packets from consumers to broker for the network issues'
similation. We ran into the similiar problem.
A few partitions were assigned to the more than one consumers.
We are using kafka 2.3.0 and StickyAssignor assignment strategy. It looks good
to us to use another assignment strategy.
It is easy to reproduce. Just adding rules to reject/drop packets from consumer
then delete the rules. We can see the following output.
{code:java}
bin/kafka-consumer-groups.sh --bootstrap-server 172.93.57.21:9092 --group
test-group --describe GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID test-group test-topic 12 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 2 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 5 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 15 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 3 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 8 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 7 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 5 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 4 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 11 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 0 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 3 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 10 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 6 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 12 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
test-group test-topic 2 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b
/172.93.57.21 consumer-1 test-group test-topic 14 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
test-group test-topic 1 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b
/172.93.57.21 consumer-1 test-group test-topic 13 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
test-group test-topic 7 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b
/172.93.57.21 consumer-1 test-group test-topic 9 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
{code}
I'm not sure if this is same problem. Could you help take a look at it? Thanks.
was (Author: redbrick9):
We are testing our application based on kafka, when we are using iptables to
drop/reject packets from consumers to broker for the network issues'
similation. We ran into the similiar problem.
A few partitions were assigned to the more than one consumers.
We are using kafka 2.3.0 and StickyAssignor assignment strategy. It looks good
to us to use another assignment strategy.
It is easy to reproduce. Just adding rules to reject/drop packets from consumer
then delete the rules. We can see the following output.
{code:java}
bin/kafka-consumer-groups.sh --bootstrap-server 172.93.57.21:9092 --group
test-group --describe GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID test-group test-topic 12 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 2 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 5 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 15 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 3 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 8 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 7 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 5 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 4 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 11 - 0 -
[jira] [Comment Edited] (KAFKA-7026) Sticky assignor could assign a partition to multiple consumers (KIP-341)
[
https://issues.apache.org/jira/browse/KAFKA-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932965#comment-16932965
]
redbrick9 edited comment on KAFKA-7026 at 9/19/19 1:19 AM:
---
We are testing our application based on kafka, when we are using iptables to
drop/reject packets from consumers to broker for the network issues'
similation. We ran into the similiar problem.
A few partitions were assigned to the more than one consumers.
We are using kafka 2.3.0 and StickyAssignor assignment strategy. It looks good
to us to use another assignment strategy.
It is easy to reproduce. Just adding rules to reject/drop packets from consumer
then delete the rules. We can see the following output.
{code:java}
bin/kafka-consumer-groups.sh --bootstrap-server 172.93.57.21:9092 --group
test-group --describe GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID test-group test-topic 12 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 2 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 5 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 15 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 3 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 8 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 7 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 5 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 4 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 11 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 0 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 3 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 10 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 6 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 12 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
test-group test-topic 2 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b
/172.93.57.21 consumer-1 test-group test-topic 14 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
test-group test-topic 1 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b
/172.93.57.21 consumer-1 test-group test-topic 13 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
test-group test-topic 7 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b
/172.93.57.21 consumer-1 test-group test-topic 9 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
{code}
was (Author: redbrick9):
We are testing our application based on kafka, when we are using iptables to
drop/reject packets from consumers to broker for the network issues'
similation. We ran into the similiar problem.
A few partitions were assigned to the more than one consumers.
We are using kafka 2.3.0 and StickyAssignor assignment strategy. It looks good
to us to use another assignment strategy.
It is easy to reproduce. Just adding rules to reject/drop packets from consumer
then delete the rules. We can see the following output.
{code:java}
// code placeholder
{code}
bin/kafka-consumer-groups.sh --bootstrap-server 172.93.57.21:9092 --group
test-group --describe GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID test-group test-topic 12 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 2 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 5 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 15 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 3 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 8 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 7 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 5 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 4 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 11 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
[jira] [Commented] (KAFKA-7026) Sticky assignor could assign a partition to multiple consumers (KIP-341)
[
https://issues.apache.org/jira/browse/KAFKA-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932965#comment-16932965
]
redbrick9 commented on KAFKA-7026:
--
We are testing our application based on kafka, when we are using iptables to
drop/reject packets from consumers to broker for the network issues'
similation. We ran into the similiar problem.
A few partitions were assigned to the more than one consumers.
We are using kafka 2.3.0 and StickyAssignor assignment strategy. It looks good
to us to use another assignment strategy.
It is easy to reproduce. Just adding rules to reject/drop packets from consumer
then delete the rules. We can see the following output.
{code:java}
// code placeholder
{code}
bin/kafka-consumer-groups.sh --bootstrap-server 172.93.57.21:9092 --group
test-group --describe GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID test-group test-topic 12 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 2 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 5 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 15 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 3 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 8 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682
/172.93.57.22 consumer-1 test-group test-topic 7 - 0 -
consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1
test-group test-topic 5 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 4 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 11 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 0 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 3 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 10 - 0 -
consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1
test-group test-topic 6 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320
/172.93.57.23 consumer-1 test-group test-topic 12 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
test-group test-topic 2 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b
/172.93.57.21 consumer-1 test-group test-topic 14 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
test-group test-topic 1 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b
/172.93.57.21 consumer-1 test-group test-topic 13 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
test-group test-topic 7 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b
/172.93.57.21 consumer-1 test-group test-topic 9 - 0 -
consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
> Sticky assignor could assign a partition to multiple consumers (KIP-341)
>
>
> Key: KAFKA-7026
> URL: https://issues.apache.org/jira/browse/KAFKA-7026
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Major
> Labels: kip
> Fix For: 2.3.0
>
>
> In the following scenario sticky assignor assigns a topic partition to two
> consumers in the group:
> # Create a topic {{test}} with a single partition
> # Start consumer {{c1}} in group {{sticky-group}} ({{c1}} becomes group
> leader and gets {{test-0}})
> # Start consumer {{c2}} in group {{sticky-group}} ({{c1}} holds onto
> {{test-0}}, {{c2}} does not get any partition)
> # Pause {{c1}} (e.g. using Java debugger) ({{c2}} becomes leader and takes
> over {{test-0}}, {{c1}} leaves the group)
> # Resume {{c1}}
> At this point both {{c1}} and {{c2}} will have {{test-0}} assigned to them.
>
> The reason is {{c1}} still has kept its previous assignment ({{test-0}}) from
> the last assignment it received from the leader (itself) and did not get the
> next round of assignments (when {{c2}} became leader) because it was paused.
> Both {{c1}} and {{c2}} enter the rebalance supplying {{test-0}} as their
> existing assignment. The sticky assignor code does not currently check and
> avoid this duplication.
>
> Note: This issue was originally reported on
> [StackOverflow|https://stackoverflow.com/questions/50761842/kafka-stickyassignor-breaking-delivery-to-single-consumer-in-the-group].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8859) Refactor Cache-level Streams Metrics
[ https://issues.apache.org/jira/browse/KAFKA-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932960#comment-16932960 ] ASF GitHub Bot commented on KAFKA-8859: --- cadonna commented on pull request #7367: KAFKA-8859: Refactor cache-level metrics URL: https://github.com/apache/kafka/pull/7367 Cache-level metrics are refactor according to KIP-444: - tag `client-id` changed to `thread-id` - name `hitRatio` changed to `hit-ratio` - made backward compatible by using streams config `built.in.metrics.version` ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Cache-level Streams Metrics > > > Key: KAFKA-8859 > URL: https://issues.apache.org/jira/browse/KAFKA-8859 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Assignee: Bruno Cadonna >Priority: Major > > Refactoring of cache-level Streams metrics according KIP-444. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8532) controller-event-thread deadlock with zk-session-expiry-handler0
[ https://issues.apache.org/jira/browse/KAFKA-8532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leibo updated KAFKA-8532: - Affects Version/s: (was: 2.3.0) > controller-event-thread deadlock with zk-session-expiry-handler0 > > > Key: KAFKA-8532 > URL: https://issues.apache.org/jira/browse/KAFKA-8532 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.1.1 >Reporter: leibo >Priority: Blocker > Attachments: js.log, js0.log, js1.log, js2.log > > > We have observed a serious deadlock between controller-event-thead and > zk-session-expirey-handle thread. When this issue occurred, it's only one way > to recovery the kafka cluster is restart kafka server. The follows is the > jstack log of controller-event-thead and zk-session-expiry-handle thread. > "zk-session-expiry-handler0" #163089 daemon prio=5 os_prio=0 > tid=0x7fcc9c01 nid=0xfb22 waiting on condition [0x7fcbb01f8000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0005ee3f7000> (a > java.util.concurrent.CountDownLatch$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) // > 等待controller-event-thread线程处理expireEvent > at > kafka.controller.KafkaController$Expire.waitUntilProcessingStarted(KafkaController.scala:1533) > at > kafka.controller.KafkaController$$anon$7.beforeInitializingSession(KafkaController.scala:173) > at > kafka.zookeeper.ZooKeeperClient.callBeforeInitializingSession(ZooKeeperClient.scala:408) > at > kafka.zookeeper.ZooKeeperClient.$anonfun$reinitialize$1(ZooKeeperClient.scala:374) > at > kafka.zookeeper.ZooKeeperClient.$anonfun$reinitialize$1$adapted(ZooKeeperClient.scala:374) > at kafka.zookeeper.ZooKeeperClient$$Lambda$1473/1823438251.apply(Unknown > Source) > at scala.collection.Iterator.foreach(Iterator.scala:937) > at scala.collection.Iterator.foreach$(Iterator.scala:937) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1425) > at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:209) > at kafka.zookeeper.ZooKeeperClient.reinitialize(ZooKeeperClient.scala:374) > at > kafka.zookeeper.ZooKeeperClient.$anonfun$scheduleSessionExpiryHandler$1(ZooKeeperClient.scala:428) > at > kafka.zookeeper.ZooKeeperClient$$Lambda$1471/701792920.apply$mcV$sp(Unknown > Source) > at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114) > at kafka.utils.KafkaScheduler$$Lambda$198/1048098469.apply$mcV$sp(Unknown > Source) > at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Locked ownable synchronizers: > - <0x000661e8d2e0> (a java.util.concurrent.ThreadPoolExecutor$Worker) > "controller-event-thread" #51 prio=5 os_prio=0 tid=0x7fceaeec4000 > nid=0x310 waiting on condition [0x7fccb55c8000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0005d1be5a00> (a > java.util.concurrent.CountDownLatch$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) > at kafka.zookeeper.ZooKeeperClient.handleRequests(ZooKeeperClient.scala:157) > at > kafka.zk.KafkaZkClient.retryRequestsUntilConnected(KafkaZkClient.scala:1596) > at >
[jira] [Issue Comment Deleted] (KAFKA-8587) EOS producer should understand consumer group assignment semantics
[ https://issues.apache.org/jira/browse/KAFKA-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noam Berman updated KAFKA-8587: --- Comment: was deleted (was: [~bchen225242] Hey, any progress on this? Really looking forward to this feature.) > EOS producer should understand consumer group assignment semantics > -- > > Key: KAFKA-8587 > URL: https://issues.apache.org/jira/browse/KAFKA-8587 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Major > > Currently exactly-once producer is coupled with individual input partitions. > This is not a well scaled solution, and the root cause is that EOS producer > doesn't understand the topic partition move throughout consumer group > rebalance. By covering this semantic gap, we could achieve much better EOS > scalability. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8922) Failed to get end offsets for topic partitions of global store
[
https://issues.apache.org/jira/browse/KAFKA-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raman Gupta resolved KAFKA-8922.
Resolution: Invalid
Closing as the error had nothing to do with streams -- just general broker
unavailability which was reported with a poor error message by the client.
Still don't know why the broker were unavailable but, hey, that's Kafka!
> Failed to get end offsets for topic partitions of global store
> --
>
> Key: KAFKA-8922
> URL: https://issues.apache.org/jira/browse/KAFKA-8922
> Project: Kafka
> Issue Type: Bug
>Reporter: Raman Gupta
>Priority: Major
>
> I have a Kafka stream that fails with this error on startup every time:
> {code}
> org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets
> for topic partitions of global store test-uiService-dlq-events-table-store
> after 0 retry attempts. You can increase the number of retries via
> configuration parameter `retries`.
> at
> org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.initialize(GlobalStreamThread.java:345)
> ~[kafka-streams-2.3.0.jar:?]
> at
> org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:270)
> ~[kafka-streams-2.3.0.jar:?]
> Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to get
> offsets by times in 30001ms
> {code}
> The stream was working fine and then this started happening.
> The stream now throws this error on every start. I am now going to attempt to
> reset the stream and delete its local state.
> I hate to say it, but Kafka Streams suck. Its problem after problem.
> UPDATE: Some more info: it appears that the brokers may have gotten into some
> kind of crazy state, for an unknown reason, and now they are just shrinking
> and expanding ISRs repeatedly. Trying to figure out the root cause of this
> craziness.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-8923) Revisit OnlineReplica state change in reassignments
Stanislav Kozlovski created KAFKA-8923: -- Summary: Revisit OnlineReplica state change in reassignments Key: KAFKA-8923 URL: https://issues.apache.org/jira/browse/KAFKA-8923 Project: Kafka Issue Type: Improvement Reporter: Stanislav Kozlovski In the replica state machine, when switching a partition to an OnlineReplica, we conditionally send a LeaderAndIsr request when the partition is available in the `partitionLeadershipInfo` structure. This happens when we switch states during a partition reassignment. It does not happen when a partition is created for the first time, as it is not present in `partitionLeadershipInfo` at that time This is a bit weird, because an OnlineReplica implies that the replica is alive, not necessarily in the ISR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8922) Failed to get end offsets for topic partitions of global store
[
https://issues.apache.org/jira/browse/KAFKA-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raman Gupta updated KAFKA-8922:
---
Description:
I have a Kafka stream that fails with this error on startup every time:
{code}
org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets for
topic partitions of global store test-uiService-dlq-events-table-store after 0
retry attempts. You can increase the number of retries via configuration
parameter `retries`.
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.initialize(GlobalStreamThread.java:345)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:270)
~[kafka-streams-2.3.0.jar:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to get
offsets by times in 30001ms
{code}
The stream was working fine and then this started happening.
The stream now throws this error on every start. I am now going to attempt to
reset the stream and delete its local state.
I hate to say it, but Kafka Streams suck. Its problem after problem.
UPDATE: Some more info: it appears that the brokers may have gotten into some
kind of crazy state, for an unknown reason, and now they are just shrinking and
expanding ISRs repeatedly. Trying to figure out the root cause of this
craziness.
was:
I have a Kafka stream that fails with this error on startup every time:
{code}
org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets for
topic partitions of global store test-uiService-dlq-events-table-store after 0
retry attempts. You can increase the number of retries via configuration
parameter `retries`.
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
~[kafka-streams-2.3.0.jar:?]
at
[jira] [Updated] (KAFKA-8922) Failed to get end offsets for topic partitions of global store
[
https://issues.apache.org/jira/browse/KAFKA-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raman Gupta updated KAFKA-8922:
---
Description:
I have a Kafka stream that fails with this error on startup every time:
{code}
org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets for
topic partitions of global store test-uiService-dlq-events-table-store after 0
retry attempts. You can increase the number of retries via configuration
parameter `retries`.
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.initialize(GlobalStreamThread.java:345)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:270)
~[kafka-streams-2.3.0.jar:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to get
offsets by times in 30001ms
{code}
The stream was working fine and then this started happening.
The stream now throws this error on every start. I am now going to attempt to
reset the stream and delete its local state.
I hate to say it, but Kafka Streams suck. Its problem after problem.
UPDATE: Some more info: it appears that the brokers may have gotten into some
kind of crazy state due to a librdkafka-based client (NodeJS). I see thousands
of logs per minute related to rebalancing that consumer
was:
I have a Kafka stream that fails with this error on startup every time:
{code}
org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets for
topic partitions of global store test-uiService-dlq-events-table-store after 0
retry attempts. You can increase the number of retries via configuration
parameter `retries`.
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
~[kafka-streams-2.3.0.jar:?]
at
[jira] [Created] (KAFKA-8922) Failed to get end offsets for topic partitions of global store
Raman Gupta created KAFKA-8922:
--
Summary: Failed to get end offsets for topic partitions of global
store
Key: KAFKA-8922
URL: https://issues.apache.org/jira/browse/KAFKA-8922
Project: Kafka
Issue Type: Bug
Reporter: Raman Gupta
I have a Kafka stream that fails with this error on startup every time:
{code}
org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets for
topic partitions of global store test-uiService-dlq-events-table-store after 0
retry attempts. You can increase the number of retries via configuration
parameter `retries`.
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.initialize(GlobalStreamThread.java:345)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:270)
~[kafka-streams-2.3.0.jar:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to get
offsets by times in 30001ms
{code}
The stream was working fine and then this started happening.
The stream now throws this error on every start. I am now going to attempt to
reset the stream and delete its local state.
I hate to say it, but Kafka Streams suck. Its problem after problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-8921) Avoid excessive info logs in the client side for incremental fetch
Zhanxiang (Patrick) Huang created KAFKA-8921:
Summary: Avoid excessive info logs in the client side for
incremental fetch
Key: KAFKA-8921
URL: https://issues.apache.org/jira/browse/KAFKA-8921
Project: Kafka
Issue Type: Improvement
Components: consumer
Reporter: Zhanxiang (Patrick) Huang
Assignee: Zhanxiang (Patrick) Huang
Currently in FetchSessionHandler::handleResponse, the following info log will
get printed out excessively when the session is evicted from the server-side
cache even though there is nothing wrong with the fetch request and client
cannot do much to improve it.
{noformat}
Node xxx was unable to process the fetch request with (sessionId=xxx,
epoch=xxx): FETCH_SESSION_ID_NOT_FOUND.
{noformat}
Moreover, when the fetch request gets throttled, the following info logs will
also get printed out, which are very misleading.
{noformat}
Node xxx sent an invalid full fetch response with ...
Node xxx sent an invalid incremental fetch response with ...
{noformat}
We should avoid logging these things in INFO level and print out more
informative logs for throttling.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-8913) Document topic based configs & ISR settings for Streams apps
[ https://issues.apache.org/jira/browse/KAFKA-8913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar resolved KAFKA-8913. --- Resolution: Fixed > Document topic based configs & ISR settings for Streams apps > > > Key: KAFKA-8913 > URL: https://issues.apache.org/jira/browse/KAFKA-8913 > Project: Kafka > Issue Type: Improvement > Components: documentation, streams >Reporter: Vinoth Chandar >Assignee: Vinoth Chandar >Priority: Major > > Noticed that it was not clear how to configure the internal topics . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8523) InsertField transformation fails when encountering tombstone event
[
https://issues.apache.org/jira/browse/KAFKA-8523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932686#comment-16932686
]
Randall Hauch commented on KAFKA-8523:
--
[~frederic.tardif], thanks for the comment.
[~gunnar.morling]'s last comment was:
{quote}
So I would suggest I simply update my PR so that it passes on tombstones
unmodified and that should be good enough.
{quote}
As mentioned above, I agree with this approach: *tombstones should be left
as-is and not modified*. Note that [~gunnar.morling] hasn't yet updated the PR,
but it will not be merged until that happens.
Second, any new configuration on any SMT would be prefixed in the connector
config with `transforms..`, and thus it would not clash with any
existing connector or SMT configuration. For example, *+_if_+* we were to add a
new SMT property named `behavior.on.null.values`, then it would be used in a
connector configuration like:
{code:java}
transforms=MyInsertField
transforms.MyInsertField.class=org.apache.kafka.connect.transforms.InsertField
transforms.MyInsertField.behavior.on.null.values=skip
{code}
*However, as mentioned above, both Gunnar and I agreed above that we don't
think a new configuration is necessary*.
> InsertField transformation fails when encountering tombstone event
> --
>
> Key: KAFKA-8523
> URL: https://issues.apache.org/jira/browse/KAFKA-8523
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Gunnar Morling
>Priority: Major
> Attachments: image-2019-09-17-15-53-44-038.png
>
>
> When applying the {{InsertField}} transformation to a tombstone event, an
> exception is raised:
> {code}
> org.apache.kafka.connect.errors.DataException: Only Map objects supported in
> absence of schema for [field insertion], found: null
> at
> org.apache.kafka.connect.transforms.util.Requirements.requireMap(Requirements.java:38)
> at
> org.apache.kafka.connect.transforms.InsertField.applySchemaless(InsertField.java:138)
> at
> org.apache.kafka.connect.transforms.InsertField.apply(InsertField.java:131)
> at
> org.apache.kafka.connect.transforms.InsertFieldTest.tombstone(InsertFieldTest.java:128)
> {code}
> AFAICS, the transform can still be made working in in this case by simply
> building up a new value map from scratch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8686) Flaky test ExampleConnectIntegrationTest#testSinkConnector
[ https://issues.apache.org/jira/browse/KAFKA-8686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch updated KAFKA-8686: - Summary: Flaky test ExampleConnectIntegrationTest#testSinkConnector (was: Flakey test ExampleConnectIntegrationTest#testSinkConnector) > Flaky test ExampleConnectIntegrationTest#testSinkConnector > -- > > Key: KAFKA-8686 > URL: https://issues.apache.org/jira/browse/KAFKA-8686 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, unit tests >Affects Versions: 2.4.0 >Reporter: Boyang Chen >Priority: Major > Labels: flaky-test > > https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/429/console > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSinkConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.13/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSinkConnector.test.stdout*20:09:20* > *20:09:20* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSinkConnector FAILED*20:09:20* java.lang.AssertionError: Condition > not met within timeout 15000. Connector tasks were not assigned a partition > each.*20:09:20* at > org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:376)*20:09:20* > at > org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:353)*20:09:20* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSinkConnector(ExampleConnectIntegrationTest.java:128) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch updated KAFKA-8555: - Component/s: KafkaConnect > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Boyang Chen >Priority: Major > Attachments: log-job139.txt, log-job141.txt, log-job23145.txt, > log-job23215.txt, log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8745) DumpLogSegments doesn't show keys, when the message is null
[
https://issues.apache.org/jira/browse/KAFKA-8745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Cheng resolved KAFKA-8745.
Fix Version/s: 2.4.0
Reviewer: Guozhang Wang
Resolution: Fixed
> DumpLogSegments doesn't show keys, when the message is null
> ---
>
> Key: KAFKA-8745
> URL: https://issues.apache.org/jira/browse/KAFKA-8745
> Project: Kafka
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.3.0
>Reporter: James Cheng
>Assignee: James Cheng
>Priority: Major
> Fix For: 2.4.0
>
>
> When DumpLogSegments encounters a message with a message key, but no message
> value, it doesn't print out the message key.
>
> {noformat}
> $ ~/kafka_2.11-2.2.0/bin/kafka-run-class.sh kafka.tools.DumpLogSegments
> --files compacted-0/.log --print-data-log
> Dumping compacted-0/.log
> Starting offset: 0
> baseOffset: 0 lastOffset: 3 count: 4 baseSequence: -1 lastSequence: -1
> producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional:
> false isControl: false position: 0 CreateTime: 1564696640073 size: 113 magic:
> 2 compresscodec: NONE crc: 206507478 isvalid: true
> | offset: 2 CreateTime: 1564696640073 keysize: 4 valuesize: -1 sequence: -1
> headerKeys: []
> | offset: 3 CreateTime: 1564696640073 keysize: 4 valuesize: -1 sequence: -1
> headerKeys: []
> {noformat}
> It should print out the message key.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932668#comment-16932668 ] Randall Hauch commented on KAFKA-8555: -- Multiple issues with flaky integration tests. > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job139.txt, log-job141.txt, log-job23145.txt, > log-job23215.txt, log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8901) Extend consumer group command to use the new Admin API to delete consumer offsets
[ https://issues.apache.org/jira/browse/KAFKA-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932665#comment-16932665 ] ASF GitHub Bot commented on KAFKA-8901: --- dajac commented on pull request #7362: KAFKA-8901; Extend consumer group command to use the new Admin API to delete consumer offsets (KIP-496) URL: https://github.com/apache/kafka/pull/7362 It add support to delete offsets in the `kafka-consumer-group`. *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Extend consumer group command to use the new Admin API to delete consumer > offsets > - > > Key: KAFKA-8901 > URL: https://issues.apache.org/jira/browse/KAFKA-8901 > Project: Kafka > Issue Type: Improvement >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > > KIP: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-496%3A+Administrative+API+to+delete+consumer+offsets]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-8391) Flaky Test RebalanceSourceConnectorsIntegrationTest#testDeleteConnector
[
https://issues.apache.org/jira/browse/KAFKA-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932663#comment-16932663
]
Randall Hauch edited comment on KAFKA-8391 at 9/18/19 4:53 PM:
---
Uploaded a tar file with the build output of 100 runs of the
RebalanceSourceConnectorsIntegrationTest, which failed 14 / 100 times. I
haven't had a chance to find a root cause, though.
was (Author: rhauch):
Uploaded a tar file with the build output of 100 runs of the
RebalanceSourceConnectorsIntegrationTest, which failed 14 / 100 times.
> Flaky Test RebalanceSourceConnectorsIntegrationTest#testDeleteConnector
> ---
>
> Key: KAFKA-8391
> URL: https://issues.apache.org/jira/browse/KAFKA-8391
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Randall Hauch
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.1.2, 2.4.0, 2.2.3
>
> Attachments: 100-gradle-builds.tar
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/4747/testReport/junit/org.apache.kafka.connect.integration/RebalanceSourceConnectorsIntegrationTest/testDeleteConnector/]
> {quote}java.lang.AssertionError: Condition not met within timeout 3.
> Connector tasks did not stop in time. at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:375) at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:352) at
> org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest.testDeleteConnector(RebalanceSourceConnectorsIntegrationTest.java:166){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8391) Flaky Test RebalanceSourceConnectorsIntegrationTest#testDeleteConnector
[
https://issues.apache.org/jira/browse/KAFKA-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932663#comment-16932663
]
Randall Hauch commented on KAFKA-8391:
--
Uploaded a tar file with the build output of 100 runs of the
RebalanceSourceConnectorsIntegrationTest, which failed 14 / 100 times.
> Flaky Test RebalanceSourceConnectorsIntegrationTest#testDeleteConnector
> ---
>
> Key: KAFKA-8391
> URL: https://issues.apache.org/jira/browse/KAFKA-8391
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Randall Hauch
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.1.2, 2.4.0, 2.2.3
>
> Attachments: 100-gradle-builds.tar
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/4747/testReport/junit/org.apache.kafka.connect.integration/RebalanceSourceConnectorsIntegrationTest/testDeleteConnector/]
> {quote}java.lang.AssertionError: Condition not met within timeout 3.
> Connector tasks did not stop in time. at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:375) at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:352) at
> org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest.testDeleteConnector(RebalanceSourceConnectorsIntegrationTest.java:166){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8391) Flaky Test RebalanceSourceConnectorsIntegrationTest#testDeleteConnector
[
https://issues.apache.org/jira/browse/KAFKA-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch updated KAFKA-8391:
-
Attachment: 100-gradle-builds.tar
> Flaky Test RebalanceSourceConnectorsIntegrationTest#testDeleteConnector
> ---
>
> Key: KAFKA-8391
> URL: https://issues.apache.org/jira/browse/KAFKA-8391
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Randall Hauch
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.1.2, 2.4.0, 2.2.3
>
> Attachments: 100-gradle-builds.tar
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/4747/testReport/junit/org.apache.kafka.connect.integration/RebalanceSourceConnectorsIntegrationTest/testDeleteConnector/]
> {quote}java.lang.AssertionError: Condition not met within timeout 3.
> Connector tasks did not stop in time. at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:375) at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:352) at
> org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest.testDeleteConnector(RebalanceSourceConnectorsIntegrationTest.java:166){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932658#comment-16932658 ] Ryanne Dolan commented on KAFKA-7500: - [~qihong] the direct passthrough described there is what MM2 currently does -- there are dedicated workers consuming from each source cluster and writing to each target cluster, without requiring a bunch of Connect clusters, and without requiring hops through an intermediate Kafka cluster. I think the confusion here is that "sink" does not imply "SinkConnector". Sink in this context is the target cluster. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Manikumar >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932652#comment-16932652 ] Guozhang Wang commented on KAFKA-8555: -- https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/24926/testReport/junit/org.apache.kafka.connect.integration/ExampleConnectIntegrationTest/testSourceConnector/ > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job139.txt, log-job141.txt, log-job23145.txt, > log-job23215.txt, log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[
https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932640#comment-16932640
]
Guozhang Wang commented on KAFKA-8555:
--
Happens again:
{code}
Stacktrace
org.apache.kafka.connect.errors.DataException: Insufficient records committed
by connector simple-conn in 3 millis. Records expected=2000, actual=1231
at
org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:191)
at
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:183)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
at
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at
[jira] [Updated] (KAFKA-8495) Make Round-robin / RangeAssignor to be "sticky" (part 5)
[ https://issues.apache.org/jira/browse/KAFKA-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-8495: --- Component/s: (was: streams) consumer > Make Round-robin / RangeAssignor to be "sticky" (part 5) > > > Key: KAFKA-8495 > URL: https://issues.apache.org/jira/browse/KAFKA-8495 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Guozhang Wang >Priority: Major > > For this new algorithm to be effective in reducing rebalance costs, it is > really expecting the plug-in assignor to be "sticky" in some way, such that > the diff of the newly-assigned-partitions and the > existing-assigned-partitions can be small, and hence only a few subset of the > total number of partitions need to be revoked / migrated at each rebalance in > practice – otherwise, we are just paying more rebalance for little benefits. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8494) Refactor Consumer#StickyAssignor and add CooperativeStickyAssignor (part 4)
[ https://issues.apache.org/jira/browse/KAFKA-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-8494: --- Component/s: (was: streams) consumer > Refactor Consumer#StickyAssignor and add CooperativeStickyAssignor (part 4) > --- > > Key: KAFKA-8494 > URL: https://issues.apache.org/jira/browse/KAFKA-8494 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Guozhang Wang >Assignee: Sophie Blee-Goldman >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932616#comment-16932616
]
Qihong Chen commented on KAFKA-7500:
Hi [~ryannedolan], Thanks for your quick response (y)(y)(y)
Now I know what "primary" means. According to blog [A look inside Kafka
MirrorMaker2|https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/] by
Renu Tewari,
{quote}In MM2 only one connect cluster is needed for all the cross-cluster
replications between a pair of datacenters. Now if we simply take a Kafka
source and sink connector and deploy them in tandem to do replication, the data
would need to hop through an intermediate Kafka cluster. MM2 avoids this
unnecessary data copying by a direct passthrough from source to sink.
{quote}
This is exactly what I want! Do you have release schedule for *SinkConnector*,
and *direct passthrough from source to sink* feature?
> MirrorMaker 2.0 (KIP-382)
> -
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
>Affects Versions: 2.4.0
>Reporter: Ryanne Dolan
>Assignee: Manikumar
>Priority: Major
> Labels: pull-request-available, ready-to-commit
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8767) Optimize StickyAssignor for Cooperative mode
[ https://issues.apache.org/jira/browse/KAFKA-8767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-8767: --- Component/s: consumer > Optimize StickyAssignor for Cooperative mode > > > Key: KAFKA-8767 > URL: https://issues.apache.org/jira/browse/KAFKA-8767 > Project: Kafka > Issue Type: Sub-task > Components: clients, consumer >Reporter: Sophie Blee-Goldman >Priority: Major > > In some rare cases, the StickyAssignor will fail to balance an assignment > without violating stickiness despite a balanced and sticky assignment being > possible. The implications of this for cooperative rebalancing are that an > unnecessary additional rebalance will be triggered. > This was seen to happen for example when each consumer is subscribed to some > random subset of all topics and all their subscriptions change to a different > random subset, as occurs in > AbstractStickyAssignorTest#testReassignmentWithRandomSubscriptionsAndChanges. > The initial assignment after the random subscription change obviously > involved migrating partitions, so following the cooperative protocol those > partitions are removed from the balanced first assignment, and a second > rebalance is triggered. In some cases, during the second rebalance the > assignor was unable to reach a balanced assignment without migrating a few > partitions, even though one must have been possible (since the first > assignment was balanced). A third rebalance was needed to reach a stable > balanced state. > Under the conditions in the previously mentioned test (between 20-40 > consumers, 10-20 topics (with 0-20 partitions) this third rebalance was > required roughly 30% of the time. Some initial improvements to the sticky > assignment logic reduced this to under 15%, but we should consider closing > this gap and optimizing the cooperative sticky assignment > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8767) Optimize StickyAssignor for Cooperative mode
[ https://issues.apache.org/jira/browse/KAFKA-8767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-8767: --- Component/s: (was: streams) clients > Optimize StickyAssignor for Cooperative mode > > > Key: KAFKA-8767 > URL: https://issues.apache.org/jira/browse/KAFKA-8767 > Project: Kafka > Issue Type: Sub-task > Components: clients >Reporter: Sophie Blee-Goldman >Priority: Major > > In some rare cases, the StickyAssignor will fail to balance an assignment > without violating stickiness despite a balanced and sticky assignment being > possible. The implications of this for cooperative rebalancing are that an > unnecessary additional rebalance will be triggered. > This was seen to happen for example when each consumer is subscribed to some > random subset of all topics and all their subscriptions change to a different > random subset, as occurs in > AbstractStickyAssignorTest#testReassignmentWithRandomSubscriptionsAndChanges. > The initial assignment after the random subscription change obviously > involved migrating partitions, so following the cooperative protocol those > partitions are removed from the balanced first assignment, and a second > rebalance is triggered. In some cases, during the second rebalance the > assignor was unable to reach a balanced assignment without migrating a few > partitions, even though one must have been possible (since the first > assignment was balanced). A third rebalance was needed to reach a stable > balanced state. > Under the conditions in the previously mentioned test (between 20-40 > consumers, 10-20 topics (with 0-20 partitions) this third rebalance was > required roughly 30% of the time. Some initial improvements to the sticky > assignment logic reduced this to under 15%, but we should consider closing > this gap and optimizing the cooperative sticky assignment > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8834) Distinguish URPs caused by reassignment plus other metrics
[ https://issues.apache.org/jira/browse/KAFKA-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932608#comment-16932608 ] ASF GitHub Bot commented on KAFKA-8834: --- viktorsomogyi commented on pull request #7361: KAFKA-8834: Add reassignment metrics (KIP-352) URL: https://github.com/apache/kafka/pull/7361 This PR contains: - added ReassigningPartitions metric - changed the semantics of the UnderReplicatedPartitions metric - added ReassignmentLag metric - added ReassignmentBytesOutPerSec metric - added ReassignmentBytesInPerSec metric - refactored the Partition class a bit to group the original, adding and removing replicas together ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Distinguish URPs caused by reassignment plus other metrics > -- > > Key: KAFKA-8834 > URL: https://issues.apache.org/jira/browse/KAFKA-8834 > Project: Kafka > Issue Type: Improvement >Reporter: Jason Gustafson >Assignee: Viktor Somogyi-Vass >Priority: Major > > This Jira tracks implementation of KIP-352: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8841) Optimize Partition.maybeIncrementLeaderHW
[ https://issues.apache.org/jira/browse/KAFKA-8841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-8841. Resolution: Fixed > Optimize Partition.maybeIncrementLeaderHW > - > > Key: KAFKA-8841 > URL: https://issues.apache.org/jira/browse/KAFKA-8841 > Project: Kafka > Issue Type: Improvement >Reporter: Jason Gustafson >Assignee: Lucas Bradstreet >Priority: Major > > We've observed a noticeable cost in `Partition.maybeIncrementLeaderHW` during > profiling. Basically just all the unnecessary collection copies and > iterations over the replica set. Since this is on the hot path for handling > follower fetches, we should probably put some effort into optimizing this. > A couple ideas: > # Don't do all those copies. > # Checking whether hw needs incrementing is not necessary unless an end > offset actually changed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8841) Optimize Partition.maybeIncrementLeaderHW
[ https://issues.apache.org/jira/browse/KAFKA-8841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932604#comment-16932604 ] ASF GitHub Bot commented on KAFKA-8841: --- hachikuji commented on pull request #7324: KAFKA-8841: reduce overhead of ReplicaManager.updateFollowerFetchState URL: https://github.com/apache/kafka/pull/7324 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Optimize Partition.maybeIncrementLeaderHW > - > > Key: KAFKA-8841 > URL: https://issues.apache.org/jira/browse/KAFKA-8841 > Project: Kafka > Issue Type: Improvement >Reporter: Jason Gustafson >Assignee: Lucas Bradstreet >Priority: Major > > We've observed a noticeable cost in `Partition.maybeIncrementLeaderHW` during > profiling. Basically just all the unnecessary collection copies and > iterations over the replica set. Since this is on the hot path for handling > follower fetches, we should probably put some effort into optimizing this. > A couple ideas: > # Don't do all those copies. > # Checking whether hw needs incrementing is not necessary unless an end > offset actually changed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8522) Tombstones can survive forever
[ https://issues.apache.org/jira/browse/KAFKA-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932580#comment-16932580 ] Jason Gustafson commented on KAFKA-8522: I think there is a bit more to this. This problem also impacts the cleaning of transaction markers, but I don't think the same approach will work. What we need to track is the timestamp when all the records from a transaction have been removed. That is when we start the timer for deletion. But this would be a different for every transaction and there is no guarantee that earlier transactions will be eligible for deletion before later ones. It all depends on the keys written in the transaction. > Tombstones can survive forever > -- > > Key: KAFKA-8522 > URL: https://issues.apache.org/jira/browse/KAFKA-8522 > Project: Kafka > Issue Type: Improvement > Components: log cleaner >Reporter: Evelyn Bayes >Priority: Minor > > This is a bit grey zone as to whether it's a "bug" but it is certainly > unintended behaviour. > > Under specific conditions tombstones effectively survive forever: > * Small amount of throughput; > * min.cleanable.dirty.ratio near or at 0; and > * Other parameters at default. > What happens is all the data continuously gets cycled into the oldest > segment. Old records get compacted away, but the new records continuously > update the timestamp of the oldest segment reseting the countdown for > deleting tombstones. > So tombstones build up in the oldest segment forever. > > While you could "fix" this by reducing the segment size, this can be > undesirable as a sudden change in throughput could cause a dangerous number > of segments to be created. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8703) Move PartitionAssignor to public API
[ https://issues.apache.org/jira/browse/KAFKA-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8703: Component/s: streams > Move PartitionAssignor to public API > > > Key: KAFKA-8703 > URL: https://issues.apache.org/jira/browse/KAFKA-8703 > Project: Kafka > Issue Type: Sub-task > Components: clients, streams >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > > Currently the PartitionAssignor, which is meant to be a pluggable interface, > sits in the internal package. It should be part of the public API, so we are > deprecating the old consumer.internal.PartitionAssignor in favor of a new > consumer.ConsumerPartitionAssignor. > We also want to take the opportunity to refactor the interface a bit, so as > to achieve an easier to evolve API moving forward > Due to the way assignors are instantiated, moving to a new > ConsumerPartitionAssignor interface will be fully compatible for most users > except those who have implemented the internal.PartitionAssignor (see > KAFKA-8704) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8494) Refactor Consumer#StickyAssignor and add CooperativeStickyAssignor (part 4)
[ https://issues.apache.org/jira/browse/KAFKA-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8494: Component/s: streams > Refactor Consumer#StickyAssignor and add CooperativeStickyAssignor (part 4) > --- > > Key: KAFKA-8494 > URL: https://issues.apache.org/jira/browse/KAFKA-8494 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Assignee: Sophie Blee-Goldman >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-4600) Consumer proceeds on when ConsumerRebalanceListener fails
[ https://issues.apache.org/jira/browse/KAFKA-4600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-4600: Component/s: streams > Consumer proceeds on when ConsumerRebalanceListener fails > - > > Key: KAFKA-4600 > URL: https://issues.apache.org/jira/browse/KAFKA-4600 > Project: Kafka > Issue Type: Sub-task > Components: consumer, streams >Affects Versions: 0.10.1.1 >Reporter: Braedon Vickers >Assignee: Guozhang Wang >Priority: Major > Fix For: 2.4.0 > > > One of the use cases for a ConsumerRebalanceListener is to load state > necessary for processing a partition when it is assigned. However, when > ConsumerRebalanceListener.onPartitionsAssigned() fails for some reason (i.e. > the state isn't loaded), the error is logged and the consumer proceeds on as > if nothing happened, happily consuming messages from the new partition. When > the state is relied upon for correct processing, this can be very bad, e.g. > data loss can occur. > It would be better if the error was propagated up so it could be dealt with > normally. At the very least the assignment should fail so the consumer > doesn't see any messages from the new partitions, and the rebalance can be > reattempted. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8496) Add system test for compatibility and upgrade path (part 6)
[ https://issues.apache.org/jira/browse/KAFKA-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8496: Component/s: streams > Add system test for compatibility and upgrade path (part 6) > --- > > Key: KAFKA-8496 > URL: https://issues.apache.org/jira/browse/KAFKA-8496 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Assignee: Bill Bejeck >Priority: Major > > tests: > * compatibility > * upgrade (follow the existing upgrade pattern) > * make sure we don't get stuck in rebalance loops (eventually get to some > agreed generation, verify we don't trigger more rebalances after agreed > generation) > acceptance: > * verify the partition assignment is correct > * verify that Streams returns to processing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8495) Make Round-robin / RangeAssignor to be "sticky" (part 5)
[ https://issues.apache.org/jira/browse/KAFKA-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8495: Component/s: streams > Make Round-robin / RangeAssignor to be "sticky" (part 5) > > > Key: KAFKA-8495 > URL: https://issues.apache.org/jira/browse/KAFKA-8495 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Priority: Major > > For this new algorithm to be effective in reducing rebalance costs, it is > really expecting the plug-in assignor to be "sticky" in some way, such that > the diff of the newly-assigned-partitions and the > existing-assigned-partitions can be small, and hence only a few subset of the > total number of partitions need to be revoked / migrated at each rebalance in > practice – otherwise, we are just paying more rebalance for little benefits. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8767) Optimize StickyAssignor for Cooperative mode
[ https://issues.apache.org/jira/browse/KAFKA-8767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8767: Component/s: streams > Optimize StickyAssignor for Cooperative mode > > > Key: KAFKA-8767 > URL: https://issues.apache.org/jira/browse/KAFKA-8767 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Sophie Blee-Goldman >Priority: Major > > In some rare cases, the StickyAssignor will fail to balance an assignment > without violating stickiness despite a balanced and sticky assignment being > possible. The implications of this for cooperative rebalancing are that an > unnecessary additional rebalance will be triggered. > This was seen to happen for example when each consumer is subscribed to some > random subset of all topics and all their subscriptions change to a different > random subset, as occurs in > AbstractStickyAssignorTest#testReassignmentWithRandomSubscriptionsAndChanges. > The initial assignment after the random subscription change obviously > involved migrating partitions, so following the cooperative protocol those > partitions are removed from the balanced first assignment, and a second > rebalance is triggered. In some cases, during the second rebalance the > assignor was unable to reach a balanced assignment without migrating a few > partitions, even though one must have been possible (since the first > assignment was balanced). A third rebalance was needed to reach a stable > balanced state. > Under the conditions in the previously mentioned test (between 20-40 > consumers, 10-20 topics (with 0-20 partitions) this third rebalance was > required roughly 30% of the time. Some initial improvements to the sticky > assignment logic reduced this to under 15%, but we should consider closing > this gap and optimizing the cooperative sticky assignment > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8492) Modify ConsumerCoordinator Algorithm with incremental protocol (part 2)
[ https://issues.apache.org/jira/browse/KAFKA-8492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8492: Component/s: streams > Modify ConsumerCoordinator Algorithm with incremental protocol (part 2) > --- > > Key: KAFKA-8492 > URL: https://issues.apache.org/jira/browse/KAFKA-8492 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Assignee: Guozhang Wang >Priority: Major > Fix For: 2.4.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8704) Add PartitionAssignor adapter for backwards compatibility
[ https://issues.apache.org/jira/browse/KAFKA-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8704: Component/s: streams > Add PartitionAssignor adapter for backwards compatibility > - > > Key: KAFKA-8704 > URL: https://issues.apache.org/jira/browse/KAFKA-8704 > Project: Kafka > Issue Type: Sub-task > Components: clients, streams >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > Fix For: 2.4.0 > > > As part of KIP-429, we are deprecating the old > consumer.internal.PartitionAssignor in favor of a [new > consumer.PartitionAssignor|https://issues.apache.org/jira/browse/KAFKA-8703] > interface that is part of the public API. > > Although the old PartitionAssignor was technically part of the internal > package, some users may have implemented it and this change will break source > compatibility for them as they would need to modify their class to implement > the new interface. The number of users affected may be small, but nonetheless > we would like to add an adapter to maintain compatibility for these users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8421) Allow consumer.poll() to return data in the middle of rebalance
[
https://issues.apache.org/jira/browse/KAFKA-8421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler updated KAFKA-8421:
Component/s: streams
> Allow consumer.poll() to return data in the middle of rebalance
> ---
>
> Key: KAFKA-8421
> URL: https://issues.apache.org/jira/browse/KAFKA-8421
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer, streams
>Reporter: Guozhang Wang
>Priority: Major
>
> With KIP-429 in place, today when a consumer is about to send join-group
> request its owned partitions may not be empty, meaning that some of its
> fetched data can still be returned. Nevertheless, today the logic is strict:
> {code}
> if (!updateAssignmentMetadataIfNeeded(timer)) {
> return ConsumerRecords.empty();
> }
> {code}
> I.e. if the consumer enters a rebalance it always returns no data.
> As an optimization, we can consider letting consumers to still return
> messages that still belong to its owned partitions even when it is within a
> rebalance, because we know it is safe that no one else would claim those
> partitions in this rebalance yet, and we can still commit offsets if, after
> this rebalance, the partitions need to be revoked then.
> One thing we need to take care though is the rebalance timeout, i.e. when
> consumer's processing those records they may not call the next poll() in time
> (think: Kafka Streams num.iterations mechanism), which may leads to consumer
> dropping out of the group during rebalance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8512) Looking into the Future: Assignor Version
[ https://issues.apache.org/jira/browse/KAFKA-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8512: Component/s: streams > Looking into the Future: Assignor Version > - > > Key: KAFKA-8512 > URL: https://issues.apache.org/jira/browse/KAFKA-8512 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Priority: Major > > I'd propose to modify the JoinGroup protocol in this KIP as well to take the > read `protocol version` from the PartitionAssignor. > And then on the broker side, when choosing the leader it will pick the one > with the highest protocol version instead of picking it "first come first > serve". > Although this change will not benefit the upgrade path at this time, in the > future if we need to upgrade the assignor again, as long as they would not > change the rebalance semantics (e.g. like we did in this KIP, from "eager" to > "cooperative") we can actually use one rolling bounce instead since as long > as there's one member on the newer version, that consumer will be picked. > For example, this can also help saving "version probing" cost on Streams as > well: suppose we augment the join group schema with `protocol version` in > Kafka version 2.3, and then with both brokers and clients being in version > 2.3+, on the first rolling bounce where subscription and assignment schema > and / or user metadata has changed, this protocol version will be bumped. On > the broker side, when receiving all member's join-group request, it will > choose the one that has the highest protocol version (also it assumes higher > versioned protocol is always backward compatible, i.e. the coordinator can > recognize lower versioned protocol as well) and select it as the leader. Then > the leader can decide, based on its received and deserialized subscription > information, how to assign partitions and how to encode the assignment > accordingly so that everyone can understand it. With this, in Streams for > example, no version probing would be needed since we are guaranteed the > leader knows everyone's version -- again it is assuming that higher versioned > protocol is always backward compatible -- and hence can successfully do the > assignment at that round. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8609) Add consumer metrics for rebalances (part 9)
[ https://issues.apache.org/jira/browse/KAFKA-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8609: Component/s: streams > Add consumer metrics for rebalances (part 9) > > > Key: KAFKA-8609 > URL: https://issues.apache.org/jira/browse/KAFKA-8609 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Sophie Blee-Goldman >Assignee: Guozhang Wang >Priority: Major > > We would like to track some additional metrics on the consumer side related > to rebalancing as part of this KIP, including > # listener callback latency > ## partitions-revoked-time-avg > ## partitions-revoked-time-max > ## partitions-assigned-time-avg > ## partitions-assigned-time-max > ## partitions-lost-time-avg > ## partitions-lost-time-max > # rebalance rate (# rebalances per day) > ## rebalance-rate-per-day -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8511) Looking into the Future: Heartbeat Communicated Protocol
[ https://issues.apache.org/jira/browse/KAFKA-8511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8511: Component/s: streams > Looking into the Future: Heartbeat Communicated Protocol > > > Key: KAFKA-8511 > URL: https://issues.apache.org/jira/browse/KAFKA-8511 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Priority: Major > > Note that KIP-429 relies on the fact that COOPERATIVE and EAGER members can > work together within the same generation as long as the leader recognize > both; this however may not be true moving forward if we add a third rebalance > protocol. One idea to resolve this in the future is that, instead of letting > the members to decide which protocol to use "locally" before sending the > join-group request, we will use Heartbeat request / response to piggy-back > the communication of the group's supported protocols and let members to rely > on that "global" information to make decisions. More specifically: > * On Heartbeat Request, we will add additional field as a list of protocols > that this member supports. > * On Heartbeat Response, we will add additional field as a single protocol > indicating which to use if the error code suggests re-joining the group. > The broker, upon receiving the heartbeat request, if the indicated supported > protocols does not contain the one it has decided to use for the up-coming > rebalance, then reply with an fatal error. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8493) Add PartitionsLost API in RebalanceListener (part 3)
[ https://issues.apache.org/jira/browse/KAFKA-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8493: Component/s: streams > Add PartitionsLost API in RebalanceListener (part 3) > > > Key: KAFKA-8493 > URL: https://issues.apache.org/jira/browse/KAFKA-8493 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Assignee: Guozhang Wang >Priority: Major > Fix For: 2.4.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8491) Bump up Consumer Protocol to v2 (part 1)
[ https://issues.apache.org/jira/browse/KAFKA-8491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8491: Component/s: streams > Bump up Consumer Protocol to v2 (part 1) > > > Key: KAFKA-8491 > URL: https://issues.apache.org/jira/browse/KAFKA-8491 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang >Assignee: Guozhang Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8902) Benchmark cooperative vs eager rebalancing
[ https://issues.apache.org/jira/browse/KAFKA-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8902: Component/s: streams > Benchmark cooperative vs eager rebalancing > -- > > Key: KAFKA-8902 > URL: https://issues.apache.org/jira/browse/KAFKA-8902 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Sophie Blee-Goldman >Assignee: John Roesler >Priority: Major > > Cause rebalance and measure: > * overall throughput > * paused time > * (also look at the metrics from > (https://issues.apache.org/jira/browse/KAFKA-8609)): > ** accumulated rebalance time > Cluster/topic sizing: > ** 10 instances > ** 100 tasks (each instance gets 10 tasks) > ** 1000 stores (each task gets 10 stores) > * standbys = [0 and 1] > Rolling bounce: > * with and without state loss > * shorter and faster than session timeout (shorter in particular should be > interesting) > Expand (from 9 to 10) > Contract (from 10 to 9) > With and without saturation: > EOS: > * with and without > Topology: > * stateful > * windowed agg > Key Parameterizations: > 1. control: no rebalances > 2. rolling without state loss faster than session timeout > 3. expand 9 to 10 > 4. contract 10 to 9 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8902) Benchmark cooperative vs eager rebalancing
[ https://issues.apache.org/jira/browse/KAFKA-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8902: Fix Version/s: 2.4.0 > Benchmark cooperative vs eager rebalancing > -- > > Key: KAFKA-8902 > URL: https://issues.apache.org/jira/browse/KAFKA-8902 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Sophie Blee-Goldman >Assignee: John Roesler >Priority: Major > Fix For: 2.4.0 > > > Cause rebalance and measure: > * overall throughput > * paused time > * (also look at the metrics from > (https://issues.apache.org/jira/browse/KAFKA-8609)): > ** accumulated rebalance time > Cluster/topic sizing: > ** 10 instances > ** 100 tasks (each instance gets 10 tasks) > ** 1000 stores (each task gets 10 stores) > * standbys = [0 and 1] > Rolling bounce: > * with and without state loss > * shorter and faster than session timeout (shorter in particular should be > interesting) > Expand (from 9 to 10) > Contract (from 10 to 9) > With and without saturation: > EOS: > * with and without > Topology: > * stateful > * windowed agg > Key Parameterizations: > 1. control: no rebalances > 2. rolling without state loss faster than session timeout > 3. expand 9 to 10 > 4. contract 10 to 9 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8510) Update StreamsPartitionAssignor to use the built-in owned partitions to achieve stickiness (part 7)
[ https://issues.apache.org/jira/browse/KAFKA-8510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler updated KAFKA-8510: Fix Version/s: 2.4.0 > Update StreamsPartitionAssignor to use the built-in owned partitions to > achieve stickiness (part 7) > --- > > Key: KAFKA-8510 > URL: https://issues.apache.org/jira/browse/KAFKA-8510 > Project: Kafka > Issue Type: Sub-task > Components: consumer, streams >Reporter: Guozhang Wang >Assignee: Sophie Blee-Goldman >Priority: Major > Fix For: 2.4.0 > > > Today this information is encoded as part of the user data bytes, we can now > remove it and leverage on the owned partitions of the protocol directly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8920) Provide a new property to enable/disable expired-token-cleaner scheduler
[ https://issues.apache.org/jira/browse/KAFKA-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932415#comment-16932415 ] ASF GitHub Bot commented on KAFKA-8920: --- bentocast commented on pull request #7360: KAFKA-8920: Add a new property to enable/disable token clean scheduler URL: https://github.com/apache/kafka/pull/7360 **Detail:** Introduce a new property to enable/disable the expired token clean scheduler Providing this new property would allow the developers to manipulate expired tokens externally by themselves, and Kafka process does not need to spend its resource to clean up the expired tokens, which is not the main feature. **The testing strategy:** ensure the value of scheduler enabling flag is based on - PasswordEncoderSecret Property is set or not (same as variable: tokenAuthEnabled) - DelegationTokenCleanSchedulerEnable Property If one of them is false, this flag must be false ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide a new property to enable/disable expired-token-cleaner scheduler > > > Key: KAFKA-8920 > URL: https://issues.apache.org/jira/browse/KAFKA-8920 > Project: Kafka > Issue Type: Wish > Components: config >Affects Versions: 2.3.0 >Reporter: Chatchavit Nitipongpun >Priority: Minor > Labels: TokenAuth > > Currently, the expired-token-cleaner scheduler is automatically started up > when the delegation.token.master.key is set. > Providing this new property to enable/disable this scheduler would allow the > developers to manipulate expired tokens externally by themselves, and Kafka > process does not need to spend its resource to clean up the expired tokens, > which is not the main feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8882) It's not possible to restart Kafka Streams using StateListener
[
https://issues.apache.org/jira/browse/KAFKA-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932295#comment-16932295
]
Jakub commented on KAFKA-8882:
--
hi [~vvcephei] ,
thanks for your answer. Here are all state transitions that I'm getting:
{code:java}
INFO: State transition from CREATED to REBALANCING
INFO: State transition from REBALANCING to RUNNING
INFO: State transition from RUNNING to ERROR
INFO: State transition from ERROR to PENDING_SHUTDOWN
{code}
After that all I see is
{code:java}
Connection to node 0 (localhost/127.0.0.1:9092) could not be established.
Broker may not be available
{code}
and once in a while
{code:java}
Metadata update failed
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node
assignment.
{code}
"Already in the pending shutdown state, wait to complete shutdown" is not
displayed.
> It's not possible to restart Kafka Streams using StateListener
> --
>
> Key: KAFKA-8882
> URL: https://issues.apache.org/jira/browse/KAFKA-8882
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.2.1
> Environment: Linux, Windows
>Reporter: Jakub
>Priority: Major
>
> Upon problems with connecting to a Kafka Cluster services using Kafka Streams
> stop working with the following error message:
> {code:java}
> Encountered the following unexpected Kafka exception during processing, this
> usually indicate Streams internal errors (...)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Aborted due to
> timeout
> (...)
> State transition from PENDING_SHUTDOWN to DEAD
> (...)
> All stream threads have died. The instance will be in error state and should
> be closed.
> {code}
>
> We tried to use a StateListener to automatically detect and work around this
> problem.
> However, this approach doesn't seem to work correctly:
> # KafkaStreams correctly transitions from status Error to Pending Shutdown,
> but then it stays in this status forever.
> # Occasionally, after registering a listener the status doesn't even change
> to Error.
>
> {code:java}
> kafkaStreams.setStateListener(new StateListener() {
> public void onChange(State stateNew, State stateOld) {
> if (stateNew == State.ERROR) {
> kafkaStreams.cleanUp();
> kafkaStreams.close();
>
> } else if (stateNew == State.PENDING_SHUTDOWN) {
>
> // this message is displayed, and then nothig else
> happens
> LOGGER.info("State is PENDING_SHUTDOWN");
>
> } else if (stateNew == State.NOT_RUNNING) {
> // it never gets here
> kafkaStreams = createKafkaStreams();
> kafkaStreams.start();
> }
> }
> });
> {code}
>
> Surprisingly, restarting KafkaStreams outside of a listener works fine.
> I'm happy to provide more details if required.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-8920) Provide a new properties to enable/disable expired-token-cleaner scheduler
[ https://issues.apache.org/jira/browse/KAFKA-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932235#comment-16932235 ] Chatchavit Nitipongpun edited comment on KAFKA-8920 at 9/18/19 9:21 AM: I have already worked on this and would create PR very soon. was (Author: bentocast): I am already work on this and would create PR very soon. > Provide a new properties to enable/disable expired-token-cleaner scheduler > -- > > Key: KAFKA-8920 > URL: https://issues.apache.org/jira/browse/KAFKA-8920 > Project: Kafka > Issue Type: Wish > Components: config >Affects Versions: 2.3.0 >Reporter: Chatchavit Nitipongpun >Priority: Minor > Labels: TokenAuth > > Currently, the expired-token-cleaner scheduler is automatically started up > when the delegation.token.master.key is set. > Providing this new property to enable/disable this scheduler would allow the > developers to manipulate expired tokens externally by themselves, and Kafka > process does not need to spend its resource to clean up the expired tokens, > which is not the main feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8920) Provide a new property to enable/disable expired-token-cleaner scheduler
[ https://issues.apache.org/jira/browse/KAFKA-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chatchavit Nitipongpun updated KAFKA-8920: -- Summary: Provide a new property to enable/disable expired-token-cleaner scheduler (was: Provide a new properties to enable/disable expired-token-cleaner scheduler) > Provide a new property to enable/disable expired-token-cleaner scheduler > > > Key: KAFKA-8920 > URL: https://issues.apache.org/jira/browse/KAFKA-8920 > Project: Kafka > Issue Type: Wish > Components: config >Affects Versions: 2.3.0 >Reporter: Chatchavit Nitipongpun >Priority: Minor > Labels: TokenAuth > > Currently, the expired-token-cleaner scheduler is automatically started up > when the delegation.token.master.key is set. > Providing this new property to enable/disable this scheduler would allow the > developers to manipulate expired tokens externally by themselves, and Kafka > process does not need to spend its resource to clean up the expired tokens, > which is not the main feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8920) Provide a new properties to enable/disable expired-token-cleaner scheduler
[ https://issues.apache.org/jira/browse/KAFKA-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932235#comment-16932235 ] Chatchavit Nitipongpun commented on KAFKA-8920: --- I am already work on this and would create PR very soon. > Provide a new properties to enable/disable expired-token-cleaner scheduler > -- > > Key: KAFKA-8920 > URL: https://issues.apache.org/jira/browse/KAFKA-8920 > Project: Kafka > Issue Type: Wish > Components: config >Affects Versions: 2.3.0 >Reporter: Chatchavit Nitipongpun >Priority: Minor > Labels: TokenAuth > > Currently, the expired-token-cleaner scheduler is automatically started up > when the delegation.token.master.key is set. > Providing this new property to enable/disable this scheduler would allow the > developers to manipulate expired tokens externally by themselves, and Kafka > process does not need to spend its resource to clean up the expired tokens, > which is not the main feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-8920) Provide a new properties to enable/disable expired-token-cleaner scheduler
[ https://issues.apache.org/jira/browse/KAFKA-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chatchavit Nitipongpun updated KAFKA-8920: -- Issue Type: Wish (was: Improvement) > Provide a new properties to enable/disable expired-token-cleaner scheduler > -- > > Key: KAFKA-8920 > URL: https://issues.apache.org/jira/browse/KAFKA-8920 > Project: Kafka > Issue Type: Wish > Components: config >Affects Versions: 2.3.0 >Reporter: Chatchavit Nitipongpun >Priority: Minor > Labels: TokenAuth > > Currently, the expired-token-cleaner scheduler is automatically started up > when the delegation.token.master.key is set. > Providing this new property to enable/disable this scheduler would allow the > developers to manipulate expired tokens externally by themselves, and Kafka > process does not need to spend its resource to clean up the expired tokens, > which is not the main feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-8920) Provide a new properties to enable/disable expired-token-cleaner scheduler
Chatchavit Nitipongpun created KAFKA-8920: - Summary: Provide a new properties to enable/disable expired-token-cleaner scheduler Key: KAFKA-8920 URL: https://issues.apache.org/jira/browse/KAFKA-8920 Project: Kafka Issue Type: Improvement Components: config Affects Versions: 2.3.0 Reporter: Chatchavit Nitipongpun Currently, the expired-token-cleaner scheduler is automatically started up when the delegation.token.master.key is set. Providing this new property to enable/disable this scheduler would allow the developers to manipulate expired tokens externally by themselves, and Kafka process does not need to spend its resource to clean up the expired tokens, which is not the main feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-8901) Extend consumer group command to use the new Admin API to delete consumer offsets
[ https://issues.apache.org/jira/browse/KAFKA-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot reassigned KAFKA-8901: -- Assignee: David Jacot > Extend consumer group command to use the new Admin API to delete consumer > offsets > - > > Key: KAFKA-8901 > URL: https://issues.apache.org/jira/browse/KAFKA-8901 > Project: Kafka > Issue Type: Improvement >Reporter: David Jacot >Assignee: David Jacot >Priority: Major > > KIP: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-496%3A+Administrative+API+to+delete+consumer+offsets]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
