[GitHub] [kafka] kowshik commented on a change in pull request #10271: KAFKA-12429: Added serdes for the default implementation of RLMM based on an internal topic as storage.

2021-04-29 Thread GitBox
kowshik commented on a change in pull request #10271: URL: https://github.com/apache/kafka/pull/10271#discussion_r622785076 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/serialization/BytesApiMessageSerde.java ## @@ -0,0 +1,68 @@ +/* +

[GitHub] [kafka] kowshik commented on a change in pull request #10271: KAFKA-12429: Added serdes for the default implementation of RLMM based on an internal topic as storage.

2021-04-29 Thread GitBox
kowshik commented on a change in pull request #10271: URL: https://github.com/apache/kafka/pull/10271#discussion_r622786068 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/serialization/RemoteLogSegmentMetadataTransform.java ## @@ -0,0 +

[GitHub] [kafka] kowshik commented on a change in pull request #10271: KAFKA-12429: Added serdes for the default implementation of RLMM based on an internal topic as storage.

2021-04-29 Thread GitBox
kowshik commented on a change in pull request #10271: URL: https://github.com/apache/kafka/pull/10271#discussion_r622786594 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/serialization/RemoteLogMetadataSerde.java ## @@ -0,0 +1,105 @@ +/

[GitHub] [kafka] kowshik commented on a change in pull request #10271: KAFKA-12429: Added serdes for the default implementation of RLMM based on an internal topic as storage.

2021-04-29 Thread GitBox
kowshik commented on a change in pull request #10271: URL: https://github.com/apache/kafka/pull/10271#discussion_r622787996 ## File path: storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentMetadata.java ## @@ -270,9 +247,9 @@ public String toS

[GitHub] [kafka] ableegoldman opened a new pull request #10609: KAFKA-12648: add NamedTopology [WIP]

2021-04-29 Thread GitBox
ableegoldman opened a new pull request #10609: URL: https://github.com/apache/kafka/pull/10609 Just the basics of adding the NamedTopology and including it in the TaskId -- still a WIP, I need to fill in some of the guts (mainly the subdirectory structure and passing the NamedTopology info

[GitHub] [kafka] ableegoldman commented on pull request #10609: KAFKA-12648: add NamedTopology [WIP]

2021-04-29 Thread GitBox
ableegoldman commented on pull request #10609: URL: https://github.com/apache/kafka/pull/10609#issuecomment-829010851 cc @wcarlson5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [kafka] showuon commented on pull request #10509: KAFKA-12464: enhance constrained sticky Assign algorithm

2021-04-29 Thread GitBox
showuon commented on pull request #10509: URL: https://github.com/apache/kafka/pull/10509#issuecomment-829025045 @ableegoldman , I've addressed all your comments, and have some code refactor. What I did are: 1. remove `sortedAllPartitions` to save memory 2. put all logic related to `g

[GitHub] [kafka] tombentley commented on pull request #9441: KAFKA-10614: Ensure group state (un)load is executed in the submitted order

2021-04-29 Thread GitBox
tombentley commented on pull request #9441: URL: https://github.com/apache/kafka/pull/9441#issuecomment-829027535 @hachikuji that's much better, thank you. I added some logging where we ignore on the remove path, hope that's OK. -- This is an automated message from the Apache Git Ser

[jira] [Created] (KAFKA-12730) A single Kerberos login failure fails all future connections from Java 9 onwards

2021-04-29 Thread Rajini Sivaram (Jira)
Rajini Sivaram created KAFKA-12730: -- Summary: A single Kerberos login failure fails all future connections from Java 9 onwards Key: KAFKA-12730 URL: https://issues.apache.org/jira/browse/KAFKA-12730

[GitHub] [kafka] showuon opened a new pull request #10610: MINOR: replace deprecated Class.newInstance() to new one

2021-04-29 Thread GitBox
showuon opened a new pull request #10610: URL: https://github.com/apache/kafka/pull/10610 replace deprecated `Clazz.newInstance()` to `clazz.getDeclaredConstructor().newInstance()` as described in official java doc [here](https://docs.oracle.com/javase/9/docs/api/java/lang/Class.html#newI

[GitHub] [kafka] rajinisivaram opened a new pull request #10611: KAFKA-12730; Avoid duplicate logout if Kerberos login fails to prevent NPE

2021-04-29 Thread GitBox
rajinisivaram opened a new pull request #10611: URL: https://github.com/apache/kafka/pull/10611 From Java 9 onwards, `LoginContext#logout()` throws an NPE if invoked multiple times due to https://bugs.openjdk.java.net/browse/JDK-8173069. KerberosLogin currently attempts logout followed by

[GitHub] [kafka] ncliang commented on a change in pull request #10563: KAFKA-12487: Add support for cooperative consumer protocol with sink connectors

2021-04-29 Thread GitBox
ncliang commented on a change in pull request #10563: URL: https://github.com/apache/kafka/pull/10563#discussion_r622851011 ## File path: connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java ## @@ -631,13 +648,31 @@ private void rewind() { }

[GitHub] [kafka] tombentley commented on a change in pull request #10605: KAFKA-12726 prevent a stuck Task.stop() from blocking subsequent Task.stops()s

2021-04-29 Thread GitBox
tombentley commented on a change in pull request #10605: URL: https://github.com/apache/kafka/pull/10605#discussion_r622876998 ## File path: connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java ## @@ -839,7 +840,22 @@ private void stopTask(ConnectorTaskId

[jira] [Commented] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping

2021-04-29 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335389#comment-17335389 ] Chris Egerton commented on KAFKA-12726: --- [~ryannedolan] can you confirm that this

[GitHub] [kafka] C0urante commented on pull request #10605: KAFKA-12726 prevent a stuck Task.stop() from blocking subsequent Task.stops()s

2021-04-29 Thread GitBox
C0urante commented on pull request #10605: URL: https://github.com/apache/kafka/pull/10605#issuecomment-829162686 Thanks for the ping @tombentley and thanks for the PR @ryannedolan! I've left a more thorough summary of my thoughts on the ticket as I'm not sure a PR is required at all at th

[GitHub] [kafka] vitojeng commented on pull request #10597: KAFKA-5876: Apply StreamsNotStartedException for Interactive Queries

2021-04-29 Thread GitBox
vitojeng commented on pull request #10597: URL: https://github.com/apache/kafka/pull/10597#issuecomment-829215736 @ableegoldman Please take a look. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[GitHub] [kafka] rajinisivaram commented on pull request #10611: KAFKA-12730; Avoid duplicate logout if Kerberos login fails to prevent NPE

2021-04-29 Thread GitBox
rajinisivaram commented on pull request #10611: URL: https://github.com/apache/kafka/pull/10611#issuecomment-829240637 @omkreddy Thanks for the review, merging to trunk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [kafka] rajinisivaram merged pull request #10611: KAFKA-12730; Avoid duplicate logout if Kerberos login fails to prevent NPE

2021-04-29 Thread GitBox
rajinisivaram merged pull request #10611: URL: https://github.com/apache/kafka/pull/10611 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [kafka] satishd commented on a change in pull request #10271: KAFKA-12429: Added serdes for the default implementation of RLMM based on an internal topic as storage.

2021-04-29 Thread GitBox
satishd commented on a change in pull request #10271: URL: https://github.com/apache/kafka/pull/10271#discussion_r623069794 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/serialization/RemoteLogMetadataSerde.java ## @@ -0,0 +1,105 @@ +/

[jira] [Created] (KAFKA-12731) High number of rebalances lead to GC Overhead limit exceeded JVM crash

2021-04-29 Thread Filip (Jira)
Filip created KAFKA-12731: - Summary: High number of rebalances lead to GC Overhead limit exceeded JVM crash Key: KAFKA-12731 URL: https://issues.apache.org/jira/browse/KAFKA-12731 Project: Kafka Iss

[jira] [Resolved] (KAFKA-12730) A single Kerberos login failure fails all future connections from Java 9 onwards

2021-04-29 Thread Rajini Sivaram (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-12730. Reviewer: Manikumar Resolution: Fixed > A single Kerberos login failure fails all fu

[jira] [Commented] (KAFKA-7260) Failures while attempting to move partitions with kafka-reassign-partitions.sh

2021-04-29 Thread Sagar Pathak (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1733#comment-1733 ] Sagar Pathak commented on KAFKA-7260: - Is this issue still open ? We are facing same

[GitHub] [kafka] satishd commented on a change in pull request #10271: KAFKA-12429: Added serdes for the default implementation of RLMM based on an internal topic as storage.

2021-04-29 Thread GitBox
satishd commented on a change in pull request #10271: URL: https://github.com/apache/kafka/pull/10271#discussion_r622766875 ## File path: storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentMetadata.java ## @@ -270,9 +247,9 @@ public String toS

[GitHub] [kafka] C0urante commented on a change in pull request #10563: KAFKA-12487: Add support for cooperative consumer protocol with sink connectors

2021-04-29 Thread GitBox
C0urante commented on a change in pull request #10563: URL: https://github.com/apache/kafka/pull/10563#discussion_r623156186 ## File path: connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java ## @@ -631,13 +648,31 @@ private void rewind() { }

[GitHub] [kafka] C0urante commented on a change in pull request #10563: KAFKA-12487: Add support for cooperative consumer protocol with sink connectors

2021-04-29 Thread GitBox
C0urante commented on a change in pull request #10563: URL: https://github.com/apache/kafka/pull/10563#discussion_r623156186 ## File path: connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java ## @@ -631,13 +648,31 @@ private void rewind() { }

[jira] [Commented] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping

2021-04-29 Thread Ryanne Dolan (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335592#comment-17335592 ] Ryanne Dolan commented on KAFKA-12726: -- [~ChrisEgerton] ah, indeed I have conflated

[jira] [Commented] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping

2021-04-29 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335638#comment-17335638 ] Chris Egerton commented on KAFKA-12726: --- Ahhh, that makes sense. We wrestled with

[jira] [Created] (KAFKA-12732) Possible Kerberos configuration bug in mirrormaker

2021-04-29 Thread Wayland Goodliffe (Jira)
Wayland Goodliffe created KAFKA-12732: - Summary: Possible Kerberos configuration bug in mirrormaker Key: KAFKA-12732 URL: https://issues.apache.org/jira/browse/KAFKA-12732 Project: Kafka

[GitHub] [kafka] hachikuji merged pull request #10480: KAFKA-12265: Move the BatchAccumulator in KafkaRaftClient to LeaderState

2021-04-29 Thread GitBox
hachikuji merged pull request #10480: URL: https://github.com/apache/kafka/pull/10480 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, plea

[jira] [Resolved] (KAFKA-12265) Move the BatchAccumulator in KafkaRaftClient to LeaderState

2021-04-29 Thread Jason Gustafson (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-12265. - Resolution: Fixed > Move the BatchAccumulator in KafkaRaftClient to LeaderState > --

[GitHub] [kafka] jolshan commented on pull request #10602: KAFKA-12724: Add 2.8.0 to system tests and streams upgrade tests.

2021-04-29 Thread GitBox
jolshan commented on pull request #10602: URL: https://github.com/apache/kafka/pull/10602#issuecomment-829412493 Can we also add some of the newer versions to`downgrade_test.py`? We've fallen behind on some of the versions there. -- This is an automated message from the Apache Git Servic

[jira] [Assigned] (KAFKA-12668) MockScheduler is not safe to use in concurrent code.

2021-04-29 Thread Maksim Iakunin (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Iakunin reassigned KAFKA-12668: -- Assignee: Maksim Iakunin > MockScheduler is not safe to use in concurrent code. > ---

[jira] [Commented] (KAFKA-12668) MockScheduler is not safe to use in concurrent code.

2021-04-29 Thread Maksim Iakunin (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335684#comment-17335684 ] Maksim Iakunin commented on KAFKA-12668: This is my first issue in the Kafka pro

[GitHub] [kafka] junrao commented on a change in pull request #10271: KAFKA-12429: Added serdes for the default implementation of RLMM based on an internal topic as storage.

2021-04-29 Thread GitBox
junrao commented on a change in pull request #10271: URL: https://github.com/apache/kafka/pull/10271#discussion_r623251580 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/serialization/BytesApiMessageSerde.java ## @@ -0,0 +1,68 @@ +/* +

[GitHub] [kafka] junrao commented on a change in pull request #10271: KAFKA-12429: Added serdes for the default implementation of RLMM based on an internal topic as storage.

2021-04-29 Thread GitBox
junrao commented on a change in pull request #10271: URL: https://github.com/apache/kafka/pull/10271#discussion_r623253824 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/serialization/RemoteLogSegmentMetadataTransform.java ## @@ -0,0 +1

[GitHub] [kafka] kowshik commented on a change in pull request #10271: KAFKA-12429: Added serdes for the default implementation of RLMM based on an internal topic as storage.

2021-04-29 Thread GitBox
kowshik commented on a change in pull request #10271: URL: https://github.com/apache/kafka/pull/10271#discussion_r623276793 ## File path: storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/serialization/RemoteLogSegmentMetadataTransform.java ## @@ -0,0 +

[GitHub] [kafka] cmccabe commented on a change in pull request #10564: MINOR: clean up some replication code

2021-04-29 Thread GitBox
cmccabe commented on a change in pull request #10564: URL: https://github.com/apache/kafka/pull/10564#discussion_r623286027 ## File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java ## @@ -356,7 +367,9 @@ public void replay(PartitionChange

[jira] [Created] (KAFKA-12733) KRaft: always bump leader epoch when changing the ISR for a controlled shutdown

2021-04-29 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-12733: Summary: KRaft: always bump leader epoch when changing the ISR for a controlled shutdown Key: KAFKA-12733 URL: https://issues.apache.org/jira/browse/KAFKA-12733 Proje

[GitHub] [kafka] cmccabe merged pull request #10564: MINOR: clean up some replication code

2021-04-29 Thread GitBox
cmccabe merged pull request #10564: URL: https://github.com/apache/kafka/pull/10564 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [kafka] wcarlson5 commented on a change in pull request #10608: MINOR: clean up some remaining locking stuff in StateDirectory

2021-04-29 Thread GitBox
wcarlson5 commented on a change in pull request #10608: URL: https://github.com/apache/kafka/pull/10608#discussion_r623295241 ## File path: streams/src/test/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImplTest.java ## @@ -152,28 +150,6 @@ public void be

[GitHub] [kafka] spena opened a new pull request #10612: KAFKA-10847: Add internal flag to disable KAFKA-10847 fix

2021-04-29 Thread GitBox
spena opened a new pull request #10612: URL: https://github.com/apache/kafka/pull/10612 Adds an internal flag that can be used to disable the fixes in KAFKA-10847. It defaults to `true` if the flag is not set or has an invalid boolean value. The flag is named `__enable.kstreams.outer

[GitHub] [kafka] C0urante commented on a change in pull request #10563: KAFKA-12487: Add support for cooperative consumer protocol with sink connectors

2021-04-29 Thread GitBox
C0urante commented on a change in pull request #10563: URL: https://github.com/apache/kafka/pull/10563#discussion_r623321728 ## File path: connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java ## @@ -680,13 +717,13 @@ public void onPartitionsAssig

[GitHub] [kafka] ableegoldman commented on a change in pull request #10608: MINOR: clean up some remaining locking stuff in StateDirectory

2021-04-29 Thread GitBox
ableegoldman commented on a change in pull request #10608: URL: https://github.com/apache/kafka/pull/10608#discussion_r623325017 ## File path: streams/src/test/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImplTest.java ## @@ -152,28 +150,6 @@ public void

[jira] [Commented] (KAFKA-9295) KTableKTableForeignKeyInnerJoinMultiIntegrationTest#shouldInnerJoinMultiPartitionQueryable

2021-04-29 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335773#comment-17335773 ] A. Sophie Blee-Goldman commented on KAFKA-9295: --- Failed again, same stacktr

[jira] [Commented] (KAFKA-7964) Flaky Test ConsumerBounceTest#testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize

2021-04-29 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335780#comment-17335780 ] A. Sophie Blee-Goldman commented on KAFKA-7964: --- Failed again, with a diffe

[GitHub] [kafka] C0urante commented on a change in pull request #10563: KAFKA-12487: Add support for cooperative consumer protocol with sink connectors

2021-04-29 Thread GitBox
C0urante commented on a change in pull request #10563: URL: https://github.com/apache/kafka/pull/10563#discussion_r623328064 ## File path: connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java ## @@ -680,13 +717,13 @@ public void onPartitionsAssig

[jira] [Commented] (KAFKA-12629) Flaky Test RaftClusterTest

2021-04-29 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335783#comment-17335783 ] A. Sophie Blee-Goldman commented on KAFKA-12629: This test has multiple

[jira] [Commented] (KAFKA-5492) LogRecoveryTest.testHWCheckpointWithFailuresSingleLogSegment transient failure

2021-04-29 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-5492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335786#comment-17335786 ] A. Sophie Blee-Goldman commented on KAFKA-5492: --- Failed again: Stacktrace

[jira] [Resolved] (KAFKA-6409) LogRecoveryTest (testHWCheckpointWithFailuresSingleLogSegment) is flaky

2021-04-29 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman resolved KAFKA-6409. --- Resolution: Duplicate > LogRecoveryTest (testHWCheckpointWithFailuresSingleLo

[GitHub] [kafka] ableegoldman commented on pull request #10608: MINOR: clean up some remaining locking stuff in StateDirectory

2021-04-29 Thread GitBox
ableegoldman commented on pull request #10608: URL: https://github.com/apache/kafka/pull/10608#issuecomment-829526743 Handful of unrelated failures (as always, these days 😞 ) -- left a comment on all their flaky test tickets -- This is an automated message from the Apache Git Service. To

[GitHub] [kafka] ryannedolan commented on pull request #10605: KAFKA-12726 prevent a stuck Task.stop() from blocking subsequent Task.stops()s

2021-04-29 Thread GitBox
ryannedolan commented on pull request #10605: URL: https://github.com/apache/kafka/pull/10605#issuecomment-829527496 @C0urante thanks for pointing out the root of our confusion here. You're right that task.stop() not being Task.stop() threw me. I believe the problem still exists tho.

[GitHub] [kafka] ableegoldman commented on pull request #10608: MINOR: clean up some remaining locking stuff in StateDirectory

2021-04-29 Thread GitBox
ableegoldman commented on pull request #10608: URL: https://github.com/apache/kafka/pull/10608#issuecomment-829528563 I'm going to merge this now since Walker approved it (thanks!) and I'm waiting to rebase the NamedTopology PR on top of these changes. If anyone comes across this and has a

[GitHub] [kafka] spena opened a new pull request #10613: KAFKA-10847: Set shared outer store to an in-memory store when in-memory stores are supplied

2021-04-29 Thread GitBox
spena opened a new pull request #10613: URL: https://github.com/apache/kafka/pull/10613 When users supply in-memory stores for left/outer joins, then the internal shared outer store must be switch to in-memory store too. This will allow users who want to keep all stores in memory to contin

[GitHub] [kafka] ableegoldman merged pull request #10608: MINOR: clean up some remaining locking stuff in StateDirectory

2021-04-29 Thread GitBox
ableegoldman merged pull request #10608: URL: https://github.com/apache/kafka/pull/10608 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, p

[GitHub] [kafka] jsancio commented on a change in pull request #10085: KAFKA-12154: Snapshot Loading API

2021-04-29 Thread GitBox
jsancio commented on a change in pull request #10085: URL: https://github.com/apache/kafka/pull/10085#discussion_r623366415 ## File path: core/src/main/scala/kafka/tools/TestRaftServer.scala ## @@ -226,7 +232,11 @@ class TestRaftServer( reader.close() }

[GitHub] [kafka] jsancio commented on a change in pull request #10085: KAFKA-12154: Snapshot Loading API

2021-04-29 Thread GitBox
jsancio commented on a change in pull request #10085: URL: https://github.com/apache/kafka/pull/10085#discussion_r623367035 ## File path: raft/src/main/java/org/apache/kafka/raft/BatchReader.java ## @@ -60,19 +61,21 @@ @Override void close(); -class Batch { +

[GitHub] [kafka] ableegoldman commented on pull request #10597: KAFKA-5876: Apply StreamsNotStartedException for Interactive Queries

2021-04-29 Thread GitBox
ableegoldman commented on pull request #10597: URL: https://github.com/apache/kafka/pull/10597#issuecomment-829564963 Thanks @vitojeng -- just to clarify, I was proposing that we do throw the StreamsNotStartedException for the `#allMetadata`, `#allMetadataForStore`, and `#queryMetadataFor

[GitHub] [kafka] jsancio commented on a change in pull request #10085: KAFKA-12154: Snapshot Loading API

2021-04-29 Thread GitBox
jsancio commented on a change in pull request #10085: URL: https://github.com/apache/kafka/pull/10085#discussion_r623367352 ## File path: raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java ## @@ -311,8 +311,18 @@ private void updateListenersProgress(long highWatermar

[GitHub] [kafka] jsancio commented on a change in pull request #10085: KAFKA-12154: Snapshot Loading API

2021-04-29 Thread GitBox
jsancio commented on a change in pull request #10085: URL: https://github.com/apache/kafka/pull/10085#discussion_r623367561 ## File path: raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java ## @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] [kafka] jsancio commented on a change in pull request #10085: KAFKA-12154: Snapshot Loading API

2021-04-29 Thread GitBox
jsancio commented on a change in pull request #10085: URL: https://github.com/apache/kafka/pull/10085#discussion_r623368881 ## File path: raft/src/test/java/org/apache/kafka/raft/RaftClientTestContext.java ## @@ -1081,24 +1103,25 @@ OptionalInt currentClaimedEpoch() {

[GitHub] [kafka] jsancio commented on a change in pull request #10085: KAFKA-12154: Snapshot Loading API

2021-04-29 Thread GitBox
jsancio commented on a change in pull request #10085: URL: https://github.com/apache/kafka/pull/10085#discussion_r623369093 ## File path: raft/src/test/java/org/apache/kafka/raft/RaftEventSimulationTest.java ## @@ -717,7 +720,8 @@ void start(int nodeId) { persi

[GitHub] [kafka] jsancio commented on a change in pull request #10085: KAFKA-12154: Snapshot Loading API

2021-04-29 Thread GitBox
jsancio commented on a change in pull request #10085: URL: https://github.com/apache/kafka/pull/10085#discussion_r623370233 ## File path: raft/src/test/java/org/apache/kafka/raft/MockLog.java ## @@ -363,33 +362,30 @@ public LogFetchInfo read(long startOffset, Isolation isolati

[GitHub] [kafka] jsancio commented on a change in pull request #10085: KAFKA-12154: Snapshot Loading API

2021-04-29 Thread GitBox
jsancio commented on a change in pull request #10085: URL: https://github.com/apache/kafka/pull/10085#discussion_r623370427 ## File path: raft/src/test/java/org/apache/kafka/raft/MockLog.java ## @@ -426,48 +422,55 @@ public RawSnapshotWriter createSnapshot(OffsetAndEpoch snaps

[GitHub] [kafka] jsancio commented on a change in pull request #10085: KAFKA-12154: Snapshot Loading API

2021-04-29 Thread GitBox
jsancio commented on a change in pull request #10085: URL: https://github.com/apache/kafka/pull/10085#discussion_r623370656 ## File path: raft/src/test/java/org/apache/kafka/raft/internals/RecordsIteratorTest.java ## @@ -0,0 +1,190 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [kafka] rondagostino commented on pull request #10604: MINOR: system test spelling/pydoc/dead code fixes

2021-04-29 Thread GitBox
rondagostino commented on pull request #10604: URL: https://github.com/apache/kafka/pull/10604#issuecomment-829572710 > Could you also update the "Co-located Raft quorum" comment in the init section with starting node id as 1? Nice catch, @kamalcph! I think I fixed what you were ref

[GitHub] [kafka] C0urante commented on a change in pull request #10563: KAFKA-12487: Add support for cooperative consumer protocol with sink connectors

2021-04-29 Thread GitBox
C0urante commented on a change in pull request #10563: URL: https://github.com/apache/kafka/pull/10563#discussion_r623409533 ## File path: connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java ## @@ -680,13 +717,13 @@ public void onPartitionsAssig

[GitHub] [kafka] Nathan22177 commented on a change in pull request #10548: KAFKA-12396 added a nullcheck before trying to retrieve a key

2021-04-29 Thread GitBox
Nathan22177 commented on a change in pull request #10548: URL: https://github.com/apache/kafka/pull/10548#discussion_r621063125 ## File path: streams/src/test/java/org/apache/kafka/streams/state/internals/MeteredSessionStoreTest.java ## @@ -472,11 +472,26 @@ public void should

[GitHub] [kafka] Nathan22177 commented on pull request #10548: KAFKA-12396 added a nullcheck before trying to retrieve a key

2021-04-29 Thread GitBox
Nathan22177 commented on pull request #10548: URL: https://github.com/apache/kafka/pull/10548#issuecomment-829635490 > To avoid checkstyle and test issues, and reduce review turn-around time, I would recommend to run unit tests and checkstyle locally before pushing update: `./gradlew clean

[GitHub] [kafka] C0urante commented on a change in pull request #10563: KAFKA-12487: Add support for cooperative consumer protocol with sink connectors

2021-04-29 Thread GitBox
C0urante commented on a change in pull request #10563: URL: https://github.com/apache/kafka/pull/10563#discussion_r623442240 ## File path: connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java ## @@ -680,13 +717,13 @@ public void onPartitionsAssig

[GitHub] [kafka] C0urante commented on a change in pull request #10563: KAFKA-12487: Add support for cooperative consumer protocol with sink connectors

2021-04-29 Thread GitBox
C0urante commented on a change in pull request #10563: URL: https://github.com/apache/kafka/pull/10563#discussion_r623442240 ## File path: connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java ## @@ -680,13 +717,13 @@ public void onPartitionsAssig

[GitHub] [kafka] ableegoldman commented on a change in pull request #10509: KAFKA-12464: enhance constrained sticky Assign algorithm

2021-04-29 Thread GitBox
ableegoldman commented on a change in pull request #10509: URL: https://github.com/apache/kafka/pull/10509#discussion_r623436340 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -163,127 +159,180 @@ private boole

[jira] [Assigned] (KAFKA-12682) Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order

2021-04-29 Thread Justine Olshan (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justine Olshan reassigned KAFKA-12682: -- Assignee: Justine Olshan > Kraft MetadataPartitionsBuilder _localChanged and _localRe

[GitHub] [kafka] vitojeng commented on pull request #10597: KAFKA-5876: Apply StreamsNotStartedException for Interactive Queries

2021-04-29 Thread GitBox
vitojeng commented on pull request #10597: URL: https://github.com/apache/kafka/pull/10597#issuecomment-829640637 Oh, I get your point. This make sense and totally agree. Thanks @ableegoldman for explanation. -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Assigned] (KAFKA-12682) Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order

2021-04-29 Thread Justine Olshan (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justine Olshan reassigned KAFKA-12682: -- Assignee: (was: Justine Olshan) > Kraft MetadataPartitionsBuilder _localChanged a

[jira] [Commented] (KAFKA-12718) SessionWindows are closed too early

2021-04-29 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17336997#comment-17336997 ] A. Sophie Blee-Goldman commented on KAFKA-12718: {quote}I was just looki

[jira] [Commented] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping

2021-04-29 Thread Ryanne Dolan (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17336550#comment-17336550 ] Ryanne Dolan commented on KAFKA-12726: -- [~ChrisEgerton] Yeah, the problem is that w

[GitHub] [kafka] xjin-Confluent opened a new pull request #10614: MINOR: Upgrade jersey to 2.34

2021-04-29 Thread GitBox
xjin-Confluent opened a new pull request #10614: URL: https://github.com/apache/kafka/pull/10614 This is to resolve CVE: CVE-2021-28168 ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status

[GitHub] [kafka] ableegoldman commented on pull request #10615: KAFKA-12648: basic skeleton API for NamedTopology

2021-04-29 Thread GitBox
ableegoldman commented on pull request #10615: URL: https://github.com/apache/kafka/pull/10615#issuecomment-829737080 @wcarlson5 @guozhangwang @rodesai -- Walker let me know if there's anything else you need, but I think this should be sufficient to unblock work on the POC while I continue

[GitHub] [kafka] lwsbox commented on pull request #5997: KAFKA-7697: Avoid blocking for leaderIsrUpdateLock in DelayedFetch

2021-04-29 Thread GitBox
lwsbox commented on pull request #5997: URL: https://github.com/apache/kafka/pull/5997#issuecomment-829737677 > Discussed with Jason and will create a new PR that avoids acquiring `Partition#leaderIsrUpdateLock` while holding the lock. which PR? -- This is an automated message fro

[GitHub] [kafka] ableegoldman opened a new pull request #10615: KAFKA-12648: basic skeleton API for NamedTopology

2021-04-29 Thread GitBox
ableegoldman opened a new pull request #10615: URL: https://github.com/apache/kafka/pull/10615 Basically just the API of https://github.com/apache/kafka/pull/10609, should not contain any logical changes in Streams at this point (for example I also stripped out the protocol change for now,

[GitHub] [kafka] ableegoldman commented on a change in pull request #10615: KAFKA-12648: basic skeleton API for NamedTopology

2021-04-29 Thread GitBox
ableegoldman commented on a change in pull request #10615: URL: https://github.com/apache/kafka/pull/10615#discussion_r623529336 ## File path: streams/src/main/java/org/apache/kafka/streams/processor/TaskId.java ## @@ -64,50 +97,98 @@ public static TaskId parse(final String tas

[jira] [Commented] (KAFKA-12718) SessionWindows are closed too early

2021-04-29 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337019#comment-17337019 ] Matthias J. Sax commented on KAFKA-12718: - Well, `gracePeriod` is a generic conc

[jira] [Commented] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping

2021-04-29 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337024#comment-17337024 ] Chris Egerton commented on KAFKA-12726: --- Okay, that makes the problem clearer (y).

[GitHub] [kafka] mjsax merged pull request #10548: KAFKA-12396 added a nullcheck before trying to retrieve a key

2021-04-29 Thread GitBox
mjsax merged pull request #10548: URL: https://github.com/apache/kafka/pull/10548 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please c

[GitHub] [kafka] mjsax commented on pull request #10548: KAFKA-12396 added a nullcheck before trying to retrieve a key

2021-04-29 Thread GitBox
mjsax commented on pull request #10548: URL: https://github.com/apache/kafka/pull/10548#issuecomment-829758387 Thanks for the PR @Nathan22177! Merged to `trunk`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [kafka] ableegoldman edited a comment on pull request #10608: MINOR: clean up some remaining locking stuff in StateDirectory

2021-04-29 Thread GitBox
ableegoldman edited a comment on pull request #10608: URL: https://github.com/apache/kafka/pull/10608#issuecomment-829528563 I'm going to merge this now since Walker approved it (thanks!) and I'm waiting to rebase the NamedTopology PR on top of these changes. If anyone comes across this an

[jira] [Created] (KAFKA-12734) LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck

2021-04-29 Thread Wenbing Shen (Jira)
Wenbing Shen created KAFKA-12734: Summary: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck Key: KAFKA-12734 URL: https://issues.apache.org/jira/browse/KAFKA-12734

[GitHub] [kafka] ableegoldman edited a comment on pull request #10608: MINOR: clean up some remaining locking stuff in StateDirectory

2021-04-29 Thread GitBox
ableegoldman edited a comment on pull request #10608: URL: https://github.com/apache/kafka/pull/10608#issuecomment-829528563 I'm going to merge this now since Walker approved it (thanks!) and I'm waiting to rebase the NamedTopology PR on top of these changes. If anyone comes across this an

[GitHub] [kafka] hachikuji opened a new pull request #10616: KAFKA-12709; Add Admin API for `ListTransactions`

2021-04-29 Thread GitBox
hachikuji opened a new pull request #10616: URL: https://github.com/apache/kafka/pull/10616 This patch adds `Admin` support for the `listTransactions` API, which was added by [KIP-664](https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+

[GitHub] [kafka] wenbingshen opened a new pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck

2021-04-29 Thread GitBox
wenbingshen opened a new pull request #10617: URL: https://github.com/apache/kafka/pull/10617 The detailed stack information is linked to this jiraId [KAFKA-12734](https://issues.apache.org/jira/browse/KAFKA-12734) ### Committer Checklist (excluded from commit message) - [ ] Verif

[GitHub] [kafka] wenbingshen commented on pull request #10617: KAFKA-12734: LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck

2021-04-29 Thread GitBox
wenbingshen commented on pull request #10617: URL: https://github.com/apache/kafka/pull/10617#issuecomment-829766268 @junrao @ijuma @chia7712 Can you take a look at this PR? Thanks. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [kafka] ableegoldman commented on pull request #6592: KAFKA-8326: Introduce List Serde

2021-04-29 Thread GitBox
ableegoldman commented on pull request #6592: URL: https://github.com/apache/kafka/pull/6592#issuecomment-829779733 Hey @yeralin one other thing, can you give the KIP document a quick pass and make sure everything in there is up to date with what we've discussed and anything else that's ev

[jira] [Commented] (KAFKA-12718) SessionWindows are closed too early

2021-04-29 Thread John Roesler (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337042#comment-17337042 ] John Roesler commented on KAFKA-12718: -- Thanks for reporting this, [~mjsax] ! What

[jira] [Assigned] (KAFKA-12734) LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when skip activeSegment sanityCheck

2021-04-29 Thread Wenbing Shen (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenbing Shen reassigned KAFKA-12734: Assignee: Wenbing Shen > LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow when

[GitHub] [kafka] feyman2016 commented on a change in pull request #10593: KAFKA-10800 Validate the snapshot id when the state machine creates a snapshot

2021-04-29 Thread GitBox
feyman2016 commented on a change in pull request #10593: URL: https://github.com/apache/kafka/pull/10593#discussion_r623566646 ## File path: raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java ## @@ -2268,6 +2269,25 @@ private Long append(int epoch, List records, bool

[GitHub] [kafka] feyman2016 commented on a change in pull request #10593: KAFKA-10800 Validate the snapshot id when the state machine creates a snapshot

2021-04-29 Thread GitBox
feyman2016 commented on a change in pull request #10593: URL: https://github.com/apache/kafka/pull/10593#discussion_r623569039 ## File path: raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java ## @@ -2268,6 +2269,25 @@ private Long append(int epoch, List records, bool

[jira] [Commented] (KAFKA-12629) Flaky Test RaftClusterTest

2021-04-29 Thread Luke Chen (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337048#comment-17337048 ] Luke Chen commented on KAFKA-12629: --- I have a PR to disable the frequently failed test

[GitHub] [kafka] dengziming opened a new pull request #10618: MINOR: Fix wrong import control declaration

2021-04-29 Thread GitBox
dengziming opened a new pull request #10618: URL: https://github.com/apache/kafka/pull/10618 *More detailed description of your change* `IntegrationTestHelper` is not right. *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integratio

[GitHub] [kafka] dengziming commented on pull request #10618: MINOR: Fix wrong import control declaration

2021-04-29 Thread GitBox
dengziming commented on pull request #10618: URL: https://github.com/apache/kafka/pull/10618#issuecomment-829787194 @mumrah ,Hello, PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [kafka] feyman2016 commented on a change in pull request #10593: KAFKA-10800 Validate the snapshot id when the state machine creates a snapshot

2021-04-29 Thread GitBox
feyman2016 commented on a change in pull request #10593: URL: https://github.com/apache/kafka/pull/10593#discussion_r623574263 ## File path: raft/src/test/java/org/apache/kafka/snapshot/SnapshotWriterTest.java ## @@ -38,7 +38,7 @@ @Test public void testWritingSnapsh

  1   2   >