[GitHub] [kafka] tombentley commented on a change in pull request #11006: KAFKA-13049: Name the threads used for log recovery

2021-07-12 Thread GitBox
tombentley commented on a change in pull request #11006: URL: https://github.com/apache/kafka/pull/11006#discussion_r668471349 ## File path: core/src/main/scala/kafka/log/LogManager.scala ## @@ -312,7 +312,15 @@ class LogManager(logDirs: Seq[File], val logDirAbsolutePath

[GitHub] [kafka] tang7526 commented on pull request #11034: KAFKA-13075: Consolidate RocksDBStore and RocksDBKeyValueStoreTest

2021-07-12 Thread GitBox
tang7526 commented on pull request #11034: URL: https://github.com/apache/kafka/pull/11034#issuecomment-878820896 @ableegoldman Could you help to review this PR? thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[GitHub] [kafka] tombentley commented on a change in pull request #11006: KAFKA-13049: Name the threads used for log recovery

2021-07-12 Thread GitBox
tombentley commented on a change in pull request #11006: URL: https://github.com/apache/kafka/pull/11006#discussion_r668466658 ## File path: core/src/main/scala/kafka/log/LogManager.scala ## @@ -312,7 +312,15 @@ class LogManager(logDirs: Seq[File], val logDirAbsolutePath

[GitHub] [kafka] tang7526 opened a new pull request #11034: KAFKA-13075: Consolidate RocksDBStore and RocksDBKeyValueStoreTest

2021-07-12 Thread GitBox
tang7526 opened a new pull request #11034: URL: https://github.com/apache/kafka/pull/11034 https://issues.apache.org/jira/browse/KAFKA-13075 ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build sta

[jira] [Assigned] (KAFKA-13075) Consolidate RocksDBStore and RocksDBKeyValueStoreTest

2021-07-12 Thread Chun-Hao Tang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chun-Hao Tang reassigned KAFKA-13075: - Assignee: Chun-Hao Tang > Consolidate RocksDBStore and RocksDBKeyValueStoreTest > -

[GitHub] [kafka] guozhangwang commented on pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment bug

2021-07-12 Thread GitBox
guozhangwang commented on pull request #10985: URL: https://github.com/apache/kafka/pull/10985#issuecomment-878781675 Just one more reply on the clearing logic, otherwise LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [kafka] guozhangwang commented on a change in pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment b

2021-07-12 Thread GitBox
guozhangwang commented on a change in pull request #10985: URL: https://github.com/apache/kafka/pull/10985#discussion_r668428677 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -205,6 +237,9 @@ private boolean a

[jira] [Created] (KAFKA-13075) Consolidate RocksDBStore and RocksDBKeyValueStoreTest

2021-07-12 Thread A. Sophie Blee-Goldman (Jira)
A. Sophie Blee-Goldman created KAFKA-13075: -- Summary: Consolidate RocksDBStore and RocksDBKeyValueStoreTest Key: KAFKA-13075 URL: https://issues.apache.org/jira/browse/KAFKA-13075 Project: Kaf

[GitHub] [kafka] guozhangwang commented on a change in pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment b

2021-07-12 Thread GitBox
guozhangwang commented on a change in pull request #10985: URL: https://github.com/apache/kafka/pull/10985#discussion_r668427821 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -218,15 +256,14 @@ private boolean

[GitHub] [kafka] satishd commented on pull request #11033: KAFKA-12988 Asynchronous API support for RemoteLogMetadataManager add/update methods.

2021-07-12 Thread GitBox
satishd commented on pull request #11033: URL: https://github.com/apache/kafka/pull/11033#issuecomment-878780060 This change is built on top of https://github.com/apache/kafka/pull/10579. The latest commit https://github.com/apache/kafka/pull/11033/commits/b7837cdc01148b5e1957bb43ed98c8f38

[jira] [Updated] (KAFKA-12925) prefixScan missing from intermediate interfaces

2021-07-12 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman updated KAFKA-12925: --- Priority: Critical (was: Major) > prefixScan missing from intermediate inte

[GitHub] [kafka] ableegoldman commented on a change in pull request #10877: KAFKA-12925: adding presfixScan operation for missed implementations

2021-07-12 Thread GitBox
ableegoldman commented on a change in pull request #10877: URL: https://github.com/apache/kafka/pull/10877#discussion_r668417652 ## File path: streams/src/main/java/org/apache/kafka/streams/state/internals/MemoryNavigableLRUCache.java ## @@ -66,6 +68,20 @@ public MemoryNavigab

[GitHub] [kafka] satishd opened a new pull request #11033: KAFKA-12988 Asynchronous API support for RemoteLogMetadataManager add/update methods.

2021-07-12 Thread GitBox
satishd opened a new pull request #11033: URL: https://github.com/apache/kafka/pull/11033 - Added asynchronous API support for RemoeLogMetadataManager add/update/put methods. - Implemented the changes on default topic based RemoteLogMetadataManager. - Refactored the respective test

[GitHub] [kafka] ableegoldman commented on pull request #10877: KAFKA-12925: adding presfixScan operation for missed implementations

2021-07-12 Thread GitBox
ableegoldman commented on pull request #10877: URL: https://github.com/apache/kafka/pull/10877#issuecomment-878765736 @vamossagar12 looks like there's a checkstyle failure that's preventing the tests from running, can you fix that as well? Giving this a pass now -- This is an automated m

[GitHub] [kafka] ableegoldman commented on pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment bug

2021-07-12 Thread GitBox
ableegoldman commented on pull request #10985: URL: https://github.com/apache/kafka/pull/10985#issuecomment-878760857 Responded to your comments @guozhangwang , let me know if that all makes sense or if you have any more concerns that need to be addressed in this PR -- This is an automat

[GitHub] [kafka] ableegoldman commented on a change in pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment b

2021-07-12 Thread GitBox
ableegoldman commented on a change in pull request #10985: URL: https://github.com/apache/kafka/pull/10985#discussion_r668410517 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -238,32 +272,50 @@ private boolean

[GitHub] [kafka] ableegoldman commented on a change in pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment b

2021-07-12 Thread GitBox
ableegoldman commented on a change in pull request #10985: URL: https://github.com/apache/kafka/pull/10985#discussion_r668409463 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -205,6 +237,9 @@ private boolean a

[GitHub] [kafka] ableegoldman commented on a change in pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment b

2021-07-12 Thread GitBox
ableegoldman commented on a change in pull request #10985: URL: https://github.com/apache/kafka/pull/10985#discussion_r668406999 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -218,15 +256,14 @@ private boolean

[GitHub] [kafka] ableegoldman commented on a change in pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment b

2021-07-12 Thread GitBox
ableegoldman commented on a change in pull request #10985: URL: https://github.com/apache/kafka/pull/10985#discussion_r668406999 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -218,15 +256,14 @@ private boolean

[jira] [Created] (KAFKA-13074) Implement mayClean for MockLog

2021-07-12 Thread Jose Armando Garcia Sancio (Jira)
Jose Armando Garcia Sancio created KAFKA-13074: -- Summary: Implement mayClean for MockLog Key: KAFKA-13074 URL: https://issues.apache.org/jira/browse/KAFKA-13074 Project: Kafka Is

[jira] [Updated] (KAFKA-13074) Implement mayClean for MockLog

2021-07-12 Thread Jose Armando Garcia Sancio (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Armando Garcia Sancio updated KAFKA-13074: --- Labels: kip-500 (was: ) > Implement mayClean for MockLog >

[GitHub] [kafka] ableegoldman commented on a change in pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment b

2021-07-12 Thread GitBox
ableegoldman commented on a change in pull request #10985: URL: https://github.com/apache/kafka/pull/10985#discussion_r668406080 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -218,15 +256,14 @@ private boolean

[jira] [Updated] (KAFKA-13073) Simulation test fails due to inconsistency in MockLog's implementation

2021-07-12 Thread Jose Armando Garcia Sancio (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Armando Garcia Sancio updated KAFKA-13073: --- Labels: kip-500 (was: ) > Simulation test fails due to inconsistenc

[GitHub] [kafka] jsancio opened a new pull request #11032: KAFKA-13073: Inconsistent MockLog implementation

2021-07-12 Thread GitBox
jsancio opened a new pull request #11032: URL: https://github.com/apache/kafka/pull/11032 Fix a simulation test failure by: 1. Relaxing the valiation of the snapshot id against the log start offset when the state machine attempts to create new snapshot. It is safe to just ignore

[GitHub] [kafka] ableegoldman commented on a change in pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment b

2021-07-12 Thread GitBox
ableegoldman commented on a change in pull request #10985: URL: https://github.com/apache/kafka/pull/10985#discussion_r668402587 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -130,19 +146,26 @@ private boolean

[GitHub] [kafka] ableegoldman commented on a change in pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment b

2021-07-12 Thread GitBox
ableegoldman commented on a change in pull request #10985: URL: https://github.com/apache/kafka/pull/10985#discussion_r668402323 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -121,7 +129,12 @@ private boolean

[jira] [Created] (KAFKA-13073) Simulation test fails due to inconsistency in MockLog's implementation

2021-07-12 Thread Jose Armando Garcia Sancio (Jira)
Jose Armando Garcia Sancio created KAFKA-13073: -- Summary: Simulation test fails due to inconsistency in MockLog's implementation Key: KAFKA-13073 URL: https://issues.apache.org/jira/browse/KAFKA-13073

[GitHub] [kafka] ableegoldman commented on pull request #10986: KAFKA-12983: reset needsJoinPrepare flag before rejoining the group

2021-07-12 Thread GitBox
ableegoldman commented on pull request #10986: URL: https://github.com/apache/kafka/pull/10986#issuecomment-878749974 Now ready for review @dajac @hachikuji @guozhangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [kafka] chia7712 commented on a change in pull request #11006: KAFKA-13049: Name the threads used for log recovery

2021-07-12 Thread GitBox
chia7712 commented on a change in pull request #11006: URL: https://github.com/apache/kafka/pull/11006#discussion_r668398858 ## File path: core/src/main/scala/kafka/log/LogManager.scala ## @@ -312,7 +312,15 @@ class LogManager(logDirs: Seq[File], val logDirAbsolutePath =

[GitHub] [kafka] ableegoldman commented on pull request #10986: KAFKA-12983: reset needsJoinPrepare flag before rejoining the group

2021-07-12 Thread GitBox
ableegoldman commented on pull request #10986: URL: https://github.com/apache/kafka/pull/10986#issuecomment-878748345 Ok I realize we actually do have a test that reproduces this already: `ConsumerCoordinatorTest.testRebalanceWithMetadataChange`. This test sets up a case where a change in

[jira] [Commented] (KAFKA-12495) Unbalanced connectors/tasks distribution will happen in Connect's incremental cooperative assignor

2021-07-12 Thread Luke Chen (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379536#comment-17379536 ] Luke Chen commented on KAFKA-12495: --- [~kkonstantine], could you take a look the PR? I

[GitHub] [kafka] showuon commented on pull request #10367: KAFKA-12495: allow consecutive revoke in incremental cooperative assignor in connector

2021-07-12 Thread GitBox
showuon commented on pull request #10367: URL: https://github.com/apache/kafka/pull/10367#issuecomment-878731886 @kkonstantine , I suddenly found this is a V3.0 blocker bug. Could you help take a look. Thanks. -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Commented] (KAFKA-12629) Failing Test: RaftClusterTest

2021-07-12 Thread Luke Chen (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379529#comment-17379529 ] Luke Chen commented on KAFKA-12629: --- After KAFKA-12677 merged into trunk in build # 3

[jira] [Created] (KAFKA-13072) refactor RemoveMembersFromConsumerGroupHandler

2021-07-12 Thread Luke Chen (Jira)
Luke Chen created KAFKA-13072: - Summary: refactor RemoveMembersFromConsumerGroupHandler Key: KAFKA-13072 URL: https://issues.apache.org/jira/browse/KAFKA-13072 Project: Kafka Issue Type: Sub-task

[jira] [Comment Edited] (KAFKA-9648) kafka server should resize backlog when create serversocket

2021-07-12 Thread Haruki Okada (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379515#comment-17379515 ] Haruki Okada edited comment on KAFKA-9648 at 7/13/21, 2:05 AM:

[jira] [Commented] (KAFKA-9648) kafka server should resize backlog when create serversocket

2021-07-12 Thread Haruki Okada (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379515#comment-17379515 ] Haruki Okada commented on KAFKA-9648: - Hi. We operate a Kafka cluster in our compa

[jira] [Commented] (KAFKA-13008) Stream will stop processing data for a long time while waiting for the partition lag

2021-07-12 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379505#comment-17379505 ] A. Sophie Blee-Goldman commented on KAFKA-13008: Thanks Guozhang, +1 on

[GitHub] [kafka] ableegoldman commented on pull request #10986: KAFKA-12983: reset needsJoinPrepare flag before rejoining the group

2021-07-12 Thread GitBox
ableegoldman commented on pull request #10986: URL: https://github.com/apache/kafka/pull/10986#issuecomment-878706682 @hachikuji in the EAGER case, after the first `onJoinPrepare` / `onPartitionsRevoked`, the subscription would have been cleared. So any subsequent invocations of `onPartiti

[GitHub] [kafka] showuon commented on pull request #10811: KAFKA-12598: remove zookeeper support on configCommand except security config

2021-07-12 Thread GitBox
showuon commented on pull request #10811: URL: https://github.com/apache/kafka/pull/10811#issuecomment-878703335 @rondagostino , thanks for the comments. Actually, I've already added 2 test cases for that: `shouldNotAllowDescribeBrokerWhileBrokerUpUsingZookeeper` -> to test `describe`

[jira] [Created] (KAFKA-13071) Deprecate and remove --authorizer option in kafka-acls.sh

2021-07-12 Thread Jason Gustafson (Jira)
Jason Gustafson created KAFKA-13071: --- Summary: Deprecate and remove --authorizer option in kafka-acls.sh Key: KAFKA-13071 URL: https://issues.apache.org/jira/browse/KAFKA-13071 Project: Kafka

[GitHub] [kafka] mjsax edited a comment on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax edited a comment on pull request #10953: URL: https://github.com/apache/kafka/pull/10953#issuecomment-878690334 > Otherwise, Streams app that run with 2.8 and before might not be compatible with Streams 3.0 because the retention time of the changelog topics created with older Streams

[GitHub] [kafka] mjsax edited a comment on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax edited a comment on pull request #10953: URL: https://github.com/apache/kafka/pull/10953#issuecomment-878687476 Correct, sliding-windows were added later and never had a default grace. Also, we deprecated `until() / maintainMs()` before we added sliding-windows and because they do no

[GitHub] [kafka] mjsax edited a comment on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax edited a comment on pull request #10953: URL: https://github.com/apache/kafka/pull/10953#issuecomment-878687476 Correct, sliding-windows were added later and never had a default grace. Also, we deprecated `until() / maintainMs()` before we added sliding-windows and because they do no

[GitHub] [kafka] mjsax edited a comment on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax edited a comment on pull request #10953: URL: https://github.com/apache/kafka/pull/10953#issuecomment-878687476 Correct, sliding-windows were added later and never had a default grace. Also, we deprecated `until() / maintainMs()` before we added sliding-windows and because they do no

[GitHub] [kafka] mjsax edited a comment on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax edited a comment on pull request #10953: URL: https://github.com/apache/kafka/pull/10953#issuecomment-878690334 > Otherwise, Streams app that run with 2.8 and before might not be compatible with Streams 3.0 because the retention time of the changelog topics created with older Streams

[GitHub] [kafka] mjsax edited a comment on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax edited a comment on pull request #10953: URL: https://github.com/apache/kafka/pull/10953#issuecomment-878690334 > Otherwise, Streams app that run with 2.8 and before might not be compatible with Streams 3.0 because the retention time of the changelog topics created with older Streams

[GitHub] [kafka] mjsax edited a comment on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax edited a comment on pull request #10953: URL: https://github.com/apache/kafka/pull/10953#issuecomment-878687476 Correct, sliding-windows were added later and never had a default grace. Also, we deprecated `until() / maintainMs()` before we added sliding-windows and because they do no

[jira] [Updated] (KAFKA-12993) Formatting of Streams 'Memory Management' docs is messed up

2021-07-12 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman updated KAFKA-12993: --- Fix Version/s: (was: 2.8.1) > Formatting of Streams 'Memory Management'

[jira] [Commented] (KAFKA-12993) Formatting of Streams 'Memory Management' docs is messed up

2021-07-12 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379486#comment-17379486 ] ASF GitHub Bot commented on KAFKA-12993: ableegoldman merged pull request #361:

[jira] [Assigned] (KAFKA-7493) Rewrite test_broker_type_bounce_at_start

2021-07-12 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman reassigned KAFKA-7493: - Assignee: (was: A. Sophie Blee-Goldman) > Rewrite test_broker_type_b

[jira] [Updated] (KAFKA-7493) Rewrite test_broker_type_bounce_at_start

2021-07-12 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman updated KAFKA-7493: -- Fix Version/s: (was: 3.0.0) 3.1.0 > Rewrite test_broker_

[GitHub] [kafka] mjsax commented on a change in pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax commented on a change in pull request #10953: URL: https://github.com/apache/kafka/pull/10953#discussion_r668344535 ## File path: streams/src/main/java/org/apache/kafka/streams/kstream/SessionWindows.java ## @@ -140,7 +140,7 @@ public static SessionWindows with(final Dur

[GitHub] [kafka] mjsax edited a comment on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax edited a comment on pull request #10953: URL: https://github.com/apache/kafka/pull/10953#issuecomment-878690334 > Otherwise, Streams app that run with 2.8 and before might not be compatible with Streams 3.0 because the retention time of the changelog topics created with older Streams

[GitHub] [kafka] mjsax commented on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax commented on pull request #10953: URL: https://github.com/apache/kafka/pull/10953#issuecomment-878690334 > Otherwise, Streams app that run with 2.8 and before might not be compatible with Streams 3.0 because the retention time of the changelog topics created with older Streams apps w

[jira] [Commented] (KAFKA-13008) Stream will stop processing data for a long time while waiting for the partition lag

2021-07-12 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379482#comment-17379482 ] Guozhang Wang commented on KAFKA-13008: --- Thanks for the great find [~showuon]! I

[GitHub] [kafka] mjsax commented on a change in pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax commented on a change in pull request #10953: URL: https://github.com/apache/kafka/pull/10953#discussion_r668344535 ## File path: streams/src/main/java/org/apache/kafka/streams/kstream/SessionWindows.java ## @@ -140,7 +140,7 @@ public static SessionWindows with(final Dur

[GitHub] [kafka] mjsax commented on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

2021-07-12 Thread GitBox
mjsax commented on pull request #10953: URL: https://github.com/apache/kafka/pull/10953#issuecomment-878687476 Correct, sliding-windows were added later and never had a default grace. Also, we deprecated `until() / maintainMs()` before and because sliding-windows do not inherit from `Windo

[GitHub] [kafka] mjsax commented on a change in pull request #10944: MINOR: Loose verification of startup in EOS system tests

2021-07-12 Thread GitBox
mjsax commented on a change in pull request #10944: URL: https://github.com/apache/kafka/pull/10944#discussion_r668337811 ## File path: tests/kafkatest/tests/streams/streams_eos_test.py ## @@ -128,45 +128,61 @@ def run_failure_and_recovery(self, processor1, processor2, process

[GitHub] [kafka] cmccabe commented on pull request #11031: KAFKA-13067 Add internal config to lower the metadata log segment size

2021-07-12 Thread GitBox
cmccabe commented on pull request #11031: URL: https://github.com/apache/kafka/pull/11031#issuecomment-878677225 Looks like a test needs to be fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [kafka] mjsax commented on a change in pull request #10994: KAFKA-8410: Update the docs to reference the new PAPI

2021-07-12 Thread GitBox
mjsax commented on a change in pull request #10994: URL: https://github.com/apache/kafka/pull/10994#discussion_r668331269 ## File path: docs/streams/developer-guide/processor-api.html ## @@ -86,12 +86,48 @@ Overviewclose() method. Note that Kafka Streams may re-use a single

[jira] [Commented] (KAFKA-13008) Stream will stop processing data for a long time while waiting for the partition lag

2021-07-12 Thread A. Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379462#comment-17379462 ] A. Sophie Blee-Goldman commented on KAFKA-13008: {quote}Re-reading KIP-2

[GitHub] [kafka] ableegoldman commented on a change in pull request #11009: MINOR: update doc for default assignor change

2021-07-12 Thread GitBox
ableegoldman commented on a change in pull request #11009: URL: https://github.com/apache/kafka/pull/11009#discussion_r668314609 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java ## @@ -114,15 +114,17 @@ "ordered by preference,

[GitHub] [kafka] mattwong949 commented on a change in pull request #10914: [KAKFA-8522] Streamline tombstone and transaction marker removal

2021-07-12 Thread GitBox
mattwong949 commented on a change in pull request #10914: URL: https://github.com/apache/kafka/pull/10914#discussion_r668302886 ## File path: clients/src/main/java/org/apache/kafka/common/record/DefaultRecordBatch.java ## @@ -246,6 +265,19 @@ public boolean isTransactional() {

[GitHub] [kafka] mattwong949 commented on a change in pull request #10914: [KAKFA-8522] Streamline tombstone and transaction marker removal

2021-07-12 Thread GitBox
mattwong949 commented on a change in pull request #10914: URL: https://github.com/apache/kafka/pull/10914#discussion_r668302886 ## File path: clients/src/main/java/org/apache/kafka/common/record/DefaultRecordBatch.java ## @@ -246,6 +265,19 @@ public boolean isTransactional() {

[GitHub] [kafka] mattwong949 commented on a change in pull request #10914: [KAKFA-8522] Streamline tombstone and transaction marker removal

2021-07-12 Thread GitBox
mattwong949 commented on a change in pull request #10914: URL: https://github.com/apache/kafka/pull/10914#discussion_r668302886 ## File path: clients/src/main/java/org/apache/kafka/common/record/DefaultRecordBatch.java ## @@ -246,6 +265,19 @@ public boolean isTransactional() {

[GitHub] [kafka] mattwong949 commented on a change in pull request #10914: [KAKFA-8522] Streamline tombstone and transaction marker removal

2021-07-12 Thread GitBox
mattwong949 commented on a change in pull request #10914: URL: https://github.com/apache/kafka/pull/10914#discussion_r668304063 ## File path: checkstyle/suppressions.xml ## @@ -57,7 +57,7 @@ Review comment: the same suppression is on L54 -- This is an a

[GitHub] [kafka] mattwong949 commented on a change in pull request #10914: [KAKFA-8522] Streamline tombstone and transaction marker removal

2021-07-12 Thread GitBox
mattwong949 commented on a change in pull request #10914: URL: https://github.com/apache/kafka/pull/10914#discussion_r668303344 ## File path: clients/src/main/java/org/apache/kafka/common/record/DefaultRecordBatch.java ## @@ -167,21 +167,9 @@ public void ensureValid() {

[GitHub] [kafka] mattwong949 commented on a change in pull request #10914: [KAKFA-8522] Streamline tombstone and transaction marker removal

2021-07-12 Thread GitBox
mattwong949 commented on a change in pull request #10914: URL: https://github.com/apache/kafka/pull/10914#discussion_r668302886 ## File path: clients/src/main/java/org/apache/kafka/common/record/DefaultRecordBatch.java ## @@ -246,6 +265,19 @@ public boolean isTransactional() {

[GitHub] [kafka] mattwong949 commented on a change in pull request #10914: [KAKFA-8522] Streamline tombstone and transaction marker removal

2021-07-12 Thread GitBox
mattwong949 commented on a change in pull request #10914: URL: https://github.com/apache/kafka/pull/10914#discussion_r668300841 ## File path: clients/src/main/java/org/apache/kafka/common/record/DefaultRecordBatch.java ## @@ -156,13 +161,27 @@ public void ensureValid() {

[GitHub] [kafka] kowshik commented on pull request #10280: KAFKA-12554: Refactor Log layer

2021-07-12 Thread GitBox
kowshik commented on pull request #10280: URL: https://github.com/apache/kafka/pull/10280#issuecomment-878641126 @junrao Thanks for the review. I ran load tests on the changes from this PR, there weren't any new regressions (i.e. latency regressions or errors) that I noticed, except for an

[jira] [Created] (KAFKA-13070) LogManager shutdown races with periodic work scheduled by the instance

2021-07-12 Thread Kowshik Prakasam (Jira)
Kowshik Prakasam created KAFKA-13070: Summary: LogManager shutdown races with periodic work scheduled by the instance Key: KAFKA-13070 URL: https://issues.apache.org/jira/browse/KAFKA-13070 Projec

[GitHub] [kafka] guozhangwang commented on a change in pull request #10985: KAFKA-12984: make AbstractStickyAssignor resilient to invalid input, utilize generation in cooperative, and fix assignment b

2021-07-12 Thread GitBox
guozhangwang commented on a change in pull request #10985: URL: https://github.com/apache/kafka/pull/10985#discussion_r668293781 ## File path: clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java ## @@ -186,16 +209,25 @@ private boolean

[GitHub] [kafka] dielhennr commented on a change in pull request #11011: KAFKA-13051: Require Principal Serde and add default

2021-07-12 Thread GitBox
dielhennr commented on a change in pull request #11011: URL: https://github.com/apache/kafka/pull/11011#discussion_r668286722 ## File path: clients/src/main/java/org/apache/kafka/common/security/authenticator/DefaultKafkaPrincipalSerde.java ## @@ -0,0 +1,51 @@ +/* + * Licensed

[jira] [Created] (KAFKA-13069) Add magic number to DefaultKafkaPrincipalBuilder.KafkaPrincipalSerde

2021-07-12 Thread Ron Dagostino (Jira)
Ron Dagostino created KAFKA-13069: - Summary: Add magic number to DefaultKafkaPrincipalBuilder.KafkaPrincipalSerde Key: KAFKA-13069 URL: https://issues.apache.org/jira/browse/KAFKA-13069 Project: Kafka

[GitHub] [kafka] d8tltanc commented on pull request #11002: KAFKA-13026: Idempotent producer (KAFKA-10619) follow-up testings

2021-07-12 Thread GitBox
d8tltanc commented on pull request #11002: URL: https://github.com/apache/kafka/pull/11002#issuecomment-878622218 @ijuma @rajinisivaram Please let me know if we are good to merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [kafka] d8tltanc commented on pull request #11002: KAFKA-13026: Idempotent producer (KAFKA-10619) follow-up testings

2021-07-12 Thread GitBox
d8tltanc commented on pull request #11002: URL: https://github.com/apache/kafka/pull/11002#issuecomment-878621846 Failed tests: Build / JDK 11 and Scala 2.13 / kafka.api.TransactionsTest.testAbortTransactionTimeout() Build / JDK 16 and Scala 2.13 / kafka.api.TransactionsTest.testSen

[GitHub] [kafka] hachikuji commented on pull request #10986: KAFKA-12983: reset needsJoinPrepare flag before rejoining the group

2021-07-12 Thread GitBox
hachikuji commented on pull request #10986: URL: https://github.com/apache/kafka/pull/10986#issuecomment-878621043 To clarify, from the perspective of the eager protocol, how would this case look? Would we get multiple calls to `onPartitionsRevoked` with the same set of partitions or somet

[GitHub] [kafka] cmccabe merged pull request #11030: MINOR: Unmarking raft quorum configs as internal

2021-07-12 Thread GitBox
cmccabe merged pull request #11030: URL: https://github.com/apache/kafka/pull/11030 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr.

[jira] [Resolved] (KAFKA-10091) Improve task idling

2021-07-12 Thread John Roesler (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-10091. -- Resolution: Fixed > Improve task idling > --- > > Key: KAFKA-1

[jira] [Resolved] (KAFKA-12360) Improve documentation of max.task.idle.ms (kafka-streams)

2021-07-12 Thread John Roesler (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-12360. -- Resolution: Fixed > Improve documentation of max.task.idle.ms (kafka-streams) > --

[jira] [Updated] (KAFKA-12977) Eliminate temporary ProducerStateManager in Log recovery logic

2021-07-12 Thread Kowshik Prakasam (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kowshik Prakasam updated KAFKA-12977: - Parent: KAFKA-12551 Issue Type: Sub-task (was: Improvement) > Eliminate tempora

[GitHub] [kafka] vvcephei commented on pull request #11003: KAFKA-12360: Document new time semantics

2021-07-12 Thread GitBox
vvcephei commented on pull request #11003: URL: https://github.com/apache/kafka/pull/11003#issuecomment-878602756 Merged and cherry-picked to 3.0 (cc @kkonstantine ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[jira] [Created] (KAFKA-13068) Rename Log to UnifiedLog

2021-07-12 Thread Kowshik Prakasam (Jira)
Kowshik Prakasam created KAFKA-13068: Summary: Rename Log to UnifiedLog Key: KAFKA-13068 URL: https://issues.apache.org/jira/browse/KAFKA-13068 Project: Kafka Issue Type: Sub-task

[jira] [Updated] (KAFKA-12554) Split Log layer into Log and LocalLog

2021-07-12 Thread Kowshik Prakasam (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kowshik Prakasam updated KAFKA-12554: - Description: Split Log layer into Log and LocalLog based on the proposal described in th

[jira] [Updated] (KAFKA-12554) Split Log layer into Log and LocalLog

2021-07-12 Thread Kowshik Prakasam (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kowshik Prakasam updated KAFKA-12554: - Summary: Split Log layer into Log and LocalLog (was: Split Log layer into UnifiedLog an

[GitHub] [kafka] vvcephei merged pull request #11003: KAFKA-12360: Document new time semantics

2021-07-12 Thread GitBox
vvcephei merged pull request #11003: URL: https://github.com/apache/kafka/pull/11003 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr

[GitHub] [kafka] vvcephei commented on pull request #11003: KAFKA-12360: Document new time semantics

2021-07-12 Thread GitBox
vvcephei commented on pull request #11003: URL: https://github.com/apache/kafka/pull/11003#issuecomment-878599816 Thanks, @JimGalasyn , @showuon , and @abbccdda ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [kafka] hachikuji commented on a change in pull request #11011: KAFKA-13051: Require Principal Serde and add default

2021-07-12 Thread GitBox
hachikuji commented on a change in pull request #11011: URL: https://github.com/apache/kafka/pull/11011#discussion_r668258168 ## File path: clients/src/main/java/org/apache/kafka/common/security/authenticator/DefaultKafkaPrincipalSerde.java ## @@ -0,0 +1,51 @@ +/* + * Licensed

[GitHub] [kafka] hachikuji commented on a change in pull request #11011: KAFKA-13051: Require Principal Serde and add default

2021-07-12 Thread GitBox
hachikuji commented on a change in pull request #11011: URL: https://github.com/apache/kafka/pull/11011#discussion_r668258168 ## File path: clients/src/main/java/org/apache/kafka/common/security/authenticator/DefaultKafkaPrincipalSerde.java ## @@ -0,0 +1,51 @@ +/* + * Licensed

[jira] [Resolved] (KAFKA-13040) Increase minimum value of segment.ms and segment.bytes

2021-07-12 Thread Badai Aqrandista (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Badai Aqrandista resolved KAFKA-13040. -- Resolution: Duplicate > Increase minimum value of segment.ms and segment.bytes > -

[jira] [Reopened] (KAFKA-7760) Add broker configuration to set minimum value for segment.bytes and segment.ms

2021-07-12 Thread Badai Aqrandista (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Badai Aqrandista reopened KAFKA-7760: - Reopening issue and making this the main ticket for KIP-760 > Add broker configuration to se

[GitHub] [kafka] guozhangwang commented on pull request #10986: KAFKA-12983: reset needsJoinPrepare flag before rejoining the group

2021-07-12 Thread GitBox
guozhangwang commented on pull request #10986: URL: https://github.com/apache/kafka/pull/10986#issuecomment-878589565 @hachikuji I think the key idea behind this fix is that, if a rebalance failed with e.g. memberId lost, then conceptually we would just started a new rebalance in which we

[GitHub] [kafka] mumrah opened a new pull request #11031: KAFKA-13067 Add internal config to lower the metadata log segment size

2021-07-12 Thread GitBox
mumrah opened a new pull request #11031: URL: https://github.com/apache/kafka/pull/11031 In order to facilitate system and integration tests that use a smaller log segment size, we are adding this internal config to lower the minimum. During normal operation, this config will use the defau

[jira] [Resolved] (KAFKA-13003) KafkaBroker advertises socket port instead of the configured advertised port

2021-07-12 Thread Uwe Eisele (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Eisele resolved KAFKA-13003. Resolution: Fixed Pull Request #10935 has been merged. > KafkaBroker advertises socket port inste

[GitHub] [kafka] cmccabe merged pull request #10935: KAFKA-13003: In kraft mode also advertise configured advertised port instead of socket port

2021-07-12 Thread GitBox
cmccabe merged pull request #10935: URL: https://github.com/apache/kafka/pull/10935 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr.

[jira] [Updated] (KAFKA-12886) Enable request forwarding by default in 3.1

2021-07-12 Thread Ryan Dielhenn (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Dielhenn updated KAFKA-12886: -- Fix Version/s: (was: 3.0.0) 3.1.0 > Enable request forwarding by defaul

[jira] [Commented] (KAFKA-12886) Enable request forwarding by default in 3.1

2021-07-12 Thread Ryan Dielhenn (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379403#comment-17379403 ] Ryan Dielhenn commented on KAFKA-12886: --- [~kkonstantine] targeting 3.1 now > Enab

[jira] [Updated] (KAFKA-12886) Enable request forwarding by default in 3.1

2021-07-12 Thread Ryan Dielhenn (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Dielhenn updated KAFKA-12886: -- Summary: Enable request forwarding by default in 3.1 (was: Enable request forwarding by defau

[GitHub] [kafka] dielhennr opened a new pull request #11030: MINOR: Unmarking raft quorum configs as internal

2021-07-12 Thread GitBox
dielhennr opened a new pull request #11030: URL: https://github.com/apache/kafka/pull/11030 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: ji

[GitHub] [kafka] showuon opened a new pull request #11026: KAFKA-13064: refactor ListConsumerGroupOffsetsHandler and tests

2021-07-12 Thread GitBox
showuon opened a new pull request #11026: URL: https://github.com/apache/kafka/pull/11026 Some issues found in the ListConsumerGroupOffsetsHandler: 1. if coordinator errors is put in the topic partition, we don't do retry 2. Didn't handle possible partition level exception T

  1   2   >