[jira] [Comment Edited] (KAFKA-9479) Describe consumer group --all-groups shows header for each entry

2020-02-17 Thread Vetle Leinonen-Roeim (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038833#comment-17038833
 ] 

Vetle Leinonen-Roeim edited comment on KAFKA-9479 at 2/18/20 7:26 AM:
--

The changes I've made will also affect when printing states in other 
situations, for instance supplying multiple groups instead of {{--all-groups. 
for instance}} {{--group group1 --group group2}}. Is this acceptable, or should 
we make an effort to only do this change for {{--all-groups}}?


was (Author: ve...@roeim.net):
The changes I've made will also affect when printing states in other 
situations, for instance supplying multiple groups instead of 
{{\-\-all-groups}}: {{\-\-group group1 \-\-group group2}}. Is this acceptable, 
or should we make an effort to only do this change for {{--all-groups}}?

> Describe consumer group --all-groups shows header for each entry
> 
>
> Key: KAFKA-9479
> URL: https://issues.apache.org/jira/browse/KAFKA-9479
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jeff Kim
>Priority: Major
>  Labels: newbie
>
> When using `bin/kafka-consumer-groups.sh --describe --state --all-groups`, we 
> print output like the following:
> {code}
> GROUP  COORDINATOR (ID)  
> ASSIGNMENT-STRATEGY  STATE   #MEMBERS
> group1 localhost:9092 (3) rangeStable  1  
>   
>
> GROUP  COORDINATOR (ID)  
> ASSIGNMENT-STRATEGY  STATE   #MEMBERS
> group2  localhost:9092 (3) rangeStable  1 
>   
>  
> {code}
> It would be nice if we did not show the header for every entry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (KAFKA-9479) Describe consumer group --all-groups shows header for each entry

2020-02-17 Thread Vetle Leinonen-Roeim (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038833#comment-17038833
 ] 

Vetle Leinonen-Roeim edited comment on KAFKA-9479 at 2/18/20 7:26 AM:
--

The changes I've made will also affect when printing states in other 
situations, for instance supplying multiple groups instead of 
{{\-\-all-groups}}, for instance {{\-\-group group1 \-\-group group2}}. Is this 
acceptable, or should we make an effort to only do this change for 
{{\-\-all-groups}}?


was (Author: ve...@roeim.net):
The changes I've made will also affect when printing states in other 
situations, for instance supplying multiple groups instead of {{--all-groups. 
for instance}} {{--group group1 --group group2}}. Is this acceptable, or should 
we make an effort to only do this change for {{--all-groups}}?

> Describe consumer group --all-groups shows header for each entry
> 
>
> Key: KAFKA-9479
> URL: https://issues.apache.org/jira/browse/KAFKA-9479
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jeff Kim
>Priority: Major
>  Labels: newbie
>
> When using `bin/kafka-consumer-groups.sh --describe --state --all-groups`, we 
> print output like the following:
> {code}
> GROUP  COORDINATOR (ID)  
> ASSIGNMENT-STRATEGY  STATE   #MEMBERS
> group1 localhost:9092 (3) rangeStable  1  
>   
>
> GROUP  COORDINATOR (ID)  
> ASSIGNMENT-STRATEGY  STATE   #MEMBERS
> group2  localhost:9092 (3) rangeStable  1 
>   
>  
> {code}
> It would be nice if we did not show the header for every entry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (KAFKA-9479) Describe consumer group --all-groups shows header for each entry

2020-02-17 Thread Vetle Leinonen-Roeim (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038833#comment-17038833
 ] 

Vetle Leinonen-Roeim edited comment on KAFKA-9479 at 2/18/20 7:09 AM:
--

The changes I've made will also affect when printing states in other 
situations, for instance supplying multiple groups instead of 
{{\-\-all-groups}}: {{\-\-group group1 \-\-group group2}}. Is this acceptable, 
or should we make an effort to only do this change for {{--all-groups}}?


was (Author: ve...@roeim.net):
The changes I've made will also affect when printing states in other 
situations, for instance supplying multiple groups instead of 
{{\-\-all-groups}}: {{\–\group group1 \-\-group group2}}. Is this acceptable, 
or should we make an effort to only do this change for {{\-\-all-groups}}?

> Describe consumer group --all-groups shows header for each entry
> 
>
> Key: KAFKA-9479
> URL: https://issues.apache.org/jira/browse/KAFKA-9479
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jeff Kim
>Priority: Major
>  Labels: newbie
>
> When using `bin/kafka-consumer-groups.sh --describe --state --all-groups`, we 
> print output like the following:
> {code}
> GROUP  COORDINATOR (ID)  
> ASSIGNMENT-STRATEGY  STATE   #MEMBERS
> group1 localhost:9092 (3) rangeStable  1  
>   
>
> GROUP  COORDINATOR (ID)  
> ASSIGNMENT-STRATEGY  STATE   #MEMBERS
> group2  localhost:9092 (3) rangeStable  1 
>   
>  
> {code}
> It would be nice if we did not show the header for every entry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (KAFKA-9479) Describe consumer group --all-groups shows header for each entry

2020-02-17 Thread Vetle Leinonen-Roeim (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038833#comment-17038833
 ] 

Vetle Leinonen-Roeim edited comment on KAFKA-9479 at 2/18/20 7:09 AM:
--

The changes I've made will also affect when printing states in other 
situations, for instance supplying multiple groups instead of 
{{\-\-all-groups}}: {{\–\group group1 \-\-group group2}}. Is this acceptable, 
or should we make an effort to only do this change for {{\-\-all-groups}}?


was (Author: ve...@roeim.net):
The changes I've made will also affect when printing states in other 
situations, for instance supplying multiple groups instead of {{--all-groups}}: 
{{–group group1 --group group2}}. Is this acceptable, or should we make an 
effort to only do this change for {{--all-groups}}?

> Describe consumer group --all-groups shows header for each entry
> 
>
> Key: KAFKA-9479
> URL: https://issues.apache.org/jira/browse/KAFKA-9479
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jeff Kim
>Priority: Major
>  Labels: newbie
>
> When using `bin/kafka-consumer-groups.sh --describe --state --all-groups`, we 
> print output like the following:
> {code}
> GROUP  COORDINATOR (ID)  
> ASSIGNMENT-STRATEGY  STATE   #MEMBERS
> group1 localhost:9092 (3) rangeStable  1  
>   
>
> GROUP  COORDINATOR (ID)  
> ASSIGNMENT-STRATEGY  STATE   #MEMBERS
> group2  localhost:9092 (3) rangeStable  1 
>   
>  
> {code}
> It would be nice if we did not show the header for every entry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9479) Describe consumer group --all-groups shows header for each entry

2020-02-17 Thread Vetle Leinonen-Roeim (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038833#comment-17038833
 ] 

Vetle Leinonen-Roeim commented on KAFKA-9479:
-

The changes I've made will also affect when printing states in other 
situations, for instance supplying multiple groups instead of {{--all-groups}}: 
{{–group group1 --group group2}}. Is this acceptable, or should we make an 
effort to only do this change for {{--all-groups}}?

> Describe consumer group --all-groups shows header for each entry
> 
>
> Key: KAFKA-9479
> URL: https://issues.apache.org/jira/browse/KAFKA-9479
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jeff Kim
>Priority: Major
>  Labels: newbie
>
> When using `bin/kafka-consumer-groups.sh --describe --state --all-groups`, we 
> print output like the following:
> {code}
> GROUP  COORDINATOR (ID)  
> ASSIGNMENT-STRATEGY  STATE   #MEMBERS
> group1 localhost:9092 (3) rangeStable  1  
>   
>
> GROUP  COORDINATOR (ID)  
> ASSIGNMENT-STRATEGY  STATE   #MEMBERS
> group2  localhost:9092 (3) rangeStable  1 
>   
>  
> {code}
> It would be nice if we did not show the header for every entry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (KAFKA-9549) Local storage implementations for RSM which can be used in tests

2020-02-17 Thread Alexandre Dupriez (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexandre Dupriez updated KAFKA-9549:
-
Summary: Local storage implementations for RSM which can be used in tests  
(was: Local storage implementations for RSM which can be used in tests.)

> Local storage implementations for RSM which can be used in tests
> 
>
> Key: KAFKA-9549
> URL: https://issues.apache.org/jira/browse/KAFKA-9549
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Satish Duggana
>Assignee: Alexandre Dupriez
>Priority: Major
>
> The goal of this task is to implement a straightforward file-system based 
> implementation of the {{RemoteStorageManager}} defined as part of the SPI for 
> Tiered Storage.
> It is intended to be used in single-host integration tests where the remote 
> storage is or can be exercised. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (KAFKA-9549) Local storage implementations for RSM which can be used in tests.

2020-02-17 Thread Alexandre Dupriez (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexandre Dupriez updated KAFKA-9549:
-
Summary: Local storage implementations for RSM which can be used in tests.  
(was: [Test Harness] Local storage implementations for RSM which can be used in 
tests.)

> Local storage implementations for RSM which can be used in tests.
> -
>
> Key: KAFKA-9549
> URL: https://issues.apache.org/jira/browse/KAFKA-9549
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Satish Duggana
>Assignee: Alexandre Dupriez
>Priority: Major
>
> The goal of this task is to implement a straightforward file-system based 
> implementation of the {{RemoteStorageManager}} defined as part of the SPI for 
> Tiered Storage.
> It is intended to be used in single-host integration tests where the remote 
> storage is or can be exercised. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (KAFKA-9564) Integration Tests for Tiered Storage

2020-02-17 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-9564:


 Summary: Integration Tests for Tiered Storage
 Key: KAFKA-9564
 URL: https://issues.apache.org/jira/browse/KAFKA-9564
 Project: Kafka
  Issue Type: Sub-task
Reporter: Alexandre Dupriez
Assignee: Alexandre Dupriez






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (KAFKA-9549) [Test Harness] Local storage implementations for RSM which can be used in tests.

2020-02-17 Thread Alexandre Dupriez (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexandre Dupriez updated KAFKA-9549:
-
Description: 
The goal of this task is to implement a straightforward file-system based 
implementation of the {{RemoteStorageManager}} defined as part of the SPI for 
Tiered Storage.

It is intended to be used in single-host integration tests where the remote 
storage is or can be exercised. 

  was:The goal of this task is to implement a straightforward file-system based 
implementation of the {{RemoteStorageManager}} defined as part of the SPI for 
Tiered Storage.


> [Test Harness] Local storage implementations for RSM which can be used in 
> tests.
> 
>
> Key: KAFKA-9549
> URL: https://issues.apache.org/jira/browse/KAFKA-9549
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Satish Duggana
>Assignee: Alexandre Dupriez
>Priority: Major
>
> The goal of this task is to implement a straightforward file-system based 
> implementation of the {{RemoteStorageManager}} defined as part of the SPI for 
> Tiered Storage.
> It is intended to be used in single-host integration tests where the remote 
> storage is or can be exercised. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9549) [Test Harness] Local storage implementations for RSM which can be used in tests.

2020-02-17 Thread Alexandre Dupriez (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038676#comment-17038676
 ] 

Alexandre Dupriez commented on KAFKA-9549:
--

[https://github.com/harshach/kafka/pull/31/files]

> [Test Harness] Local storage implementations for RSM which can be used in 
> tests.
> 
>
> Key: KAFKA-9549
> URL: https://issues.apache.org/jira/browse/KAFKA-9549
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Satish Duggana
>Assignee: Alexandre Dupriez
>Priority: Major
>
> The goal of this task is to implement a straightforward file-system based 
> implementation of the {{RemoteStorageManager}} defined as part of the SPI for 
> Tiered Storage.
> It is intended to be used in single-host integration tests where the remote 
> storage is or can be exercised. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (KAFKA-9549) [Test Harness] Local storage implementations for RSM which can be used in tests.

2020-02-17 Thread Alexandre Dupriez (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexandre Dupriez updated KAFKA-9549:
-
Description: The goal of this task is to implement a straightforward 
file-system based implementation of the {{RemoteStorageManager}} defined as 
part of the SPI for Tiered Storage.

> [Test Harness] Local storage implementations for RSM which can be used in 
> tests.
> 
>
> Key: KAFKA-9549
> URL: https://issues.apache.org/jira/browse/KAFKA-9549
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Satish Duggana
>Assignee: Alexandre Dupriez
>Priority: Major
>
> The goal of this task is to implement a straightforward file-system based 
> implementation of the {{RemoteStorageManager}} defined as part of the SPI for 
> Tiered Storage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9319) Run some system tests using TLSv1.3

2020-02-17 Thread Nikolay Izhikov (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038587#comment-17038587
 ] 

Nikolay Izhikov commented on KAFKA-9319:


[~rsivaram] TLSv1.3 was added to the java11.

Please, see, JDK issue [1].
If we enabled TLSv1.3 by default then we should require from users that kafka 
runs on JDK11.
Is it OK?

[1] https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8145252

> Run some system tests using TLSv1.3
> ---
>
> Key: KAFKA-9319
> URL: https://issues.apache.org/jira/browse/KAFKA-9319
> Project: Kafka
>  Issue Type: Test
>Reporter: Rajini Sivaram
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.5.0
>
>
> KAFKA-7251 enables TLSv1.3 for Kafka. We should get some system tests to run 
> using TLSv1.3. Since TLSv1.3 is only supported from Java 11 onwards, we need 
> a system test build that runs with JDK 11 to enable these tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Closed] (KAFKA-2435) More optimally balanced partition assignment strategy

2020-02-17 Thread Andrew Olson (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Olson closed KAFKA-2435.
---

> More optimally balanced partition assignment strategy
> -
>
> Key: KAFKA-2435
> URL: https://issues.apache.org/jira/browse/KAFKA-2435
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Andrew Olson
>Assignee: Andrew Olson
>Priority: Major
> Attachments: KAFKA-2435.patch
>
>
> While the roundrobin partition assignment strategy is an improvement over the 
> range strategy, when the consumer topic subscriptions are not identical 
> (previously disallowed but will be possible as of KAFKA-2172) it can produce 
> heavily skewed assignments. As suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767]
>  it would be nice to have a strategy that attempts to assign an equal number 
> of partitions to each consumer in a group, regardless of how similar their 
> individual topic subscriptions are. We can accomplish this by tracking the 
> number of partitions assigned to each consumer, and having the partition 
> assignment loop assign each partition to a consumer interested in that topic 
> with the least number of partitions assigned. 
> Additionally, we can optimize the distribution fairness by adjusting the 
> partition assignment order:
> * Topics with fewer consumers are assigned first.
> * In the event of a tie for least consumers, the topic with more partitions 
> is assigned first.
> The general idea behind these two rules is to keep the most flexible 
> assignment choices available as long as possible by starting with the most 
> constrained partitions/consumers.
> This JIRA addresses the original high-level consumer. For the new consumer, 
> see KAFKA-3297.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Closed] (KAFKA-3297) More optimally balanced partition assignment strategy (new consumer)

2020-02-17 Thread Andrew Olson (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Olson closed KAFKA-3297.
---

> More optimally balanced partition assignment strategy (new consumer)
> 
>
> Key: KAFKA-3297
> URL: https://issues.apache.org/jira/browse/KAFKA-3297
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Andrew Olson
>Assignee: Andrew Olson
>Priority: Major
>
> While the roundrobin partition assignment strategy is an improvement over the 
> range strategy, when the consumer topic subscriptions are not identical 
> (previously disallowed but will be possible as of KAFKA-2172) it can produce 
> heavily skewed assignments. As suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767]
>  it would be nice to have a strategy that attempts to assign an equal number 
> of partitions to each consumer in a group, regardless of how similar their 
> individual topic subscriptions are. We can accomplish this by tracking the 
> number of partitions assigned to each consumer, and having the partition 
> assignment loop assign each partition to a consumer interested in that topic 
> with the least number of partitions assigned. 
> Additionally, we can optimize the distribution fairness by adjusting the 
> partition assignment order:
> * Topics with fewer consumers are assigned first.
> * In the event of a tie for least consumers, the topic with more partitions 
> is assigned first.
> The general idea behind these two rules is to keep the most flexible 
> assignment choices available as long as possible by starting with the most 
> constrained partitions/consumers.
> This JIRA addresses the new consumer. For the original high-level consumer, 
> see KAFKA-2435.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-3297) More optimally balanced partition assignment strategy (new consumer)

2020-02-17 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038549#comment-17038549
 ] 

ASF GitHub Bot commented on KAFKA-3297:
---

noslowerdna commented on pull request #979: KAFKA-3297: Fair consumer partition 
assignment strategy (new consumer)
URL: https://github.com/apache/kafka/pull/979
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> More optimally balanced partition assignment strategy (new consumer)
> 
>
> Key: KAFKA-3297
> URL: https://issues.apache.org/jira/browse/KAFKA-3297
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Andrew Olson
>Assignee: Andrew Olson
>Priority: Major
>
> While the roundrobin partition assignment strategy is an improvement over the 
> range strategy, when the consumer topic subscriptions are not identical 
> (previously disallowed but will be possible as of KAFKA-2172) it can produce 
> heavily skewed assignments. As suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767]
>  it would be nice to have a strategy that attempts to assign an equal number 
> of partitions to each consumer in a group, regardless of how similar their 
> individual topic subscriptions are. We can accomplish this by tracking the 
> number of partitions assigned to each consumer, and having the partition 
> assignment loop assign each partition to a consumer interested in that topic 
> with the least number of partitions assigned. 
> Additionally, we can optimize the distribution fairness by adjusting the 
> partition assignment order:
> * Topics with fewer consumers are assigned first.
> * In the event of a tie for least consumers, the topic with more partitions 
> is assigned first.
> The general idea behind these two rules is to keep the most flexible 
> assignment choices available as long as possible by starting with the most 
> constrained partitions/consumers.
> This JIRA addresses the new consumer. For the original high-level consumer, 
> see KAFKA-2435.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-8245) Flaky Test DeleteConsumerGroupsTest#testDeleteCmdAllGroups

2020-02-17 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038541#comment-17038541
 ] 

ASF GitHub Bot commented on KAFKA-8245:
---

vahidhashemian commented on pull request #8032: KAFKA-8245 Flaky Test 
DeleteConsumerGroupsTest#testDeleteCmdAllGroups
URL: https://github.com/apache/kafka/pull/8032
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Flaky Test DeleteConsumerGroupsTest#testDeleteCmdAllGroups
> --
>
> Key: KAFKA-8245
> URL: https://issues.apache.org/jira/browse/KAFKA-8245
> Project: Kafka
>  Issue Type: Bug
>  Components: admin, unit tests
>Affects Versions: 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Chia-Ping Tsai
>Priority: Critical
>  Labels: flaky-test
> Fix For: 2.5.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/3781/testReport/junit/kafka.admin/DeleteConsumerGroupsTest/testDeleteCmdAllGroups/]
> {quote}java.lang.AssertionError: The group did become empty as expected. at 
> kafka.utils.TestUtils$.fail(TestUtils.scala:381) at 
> kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:791) at 
> kafka.admin.DeleteConsumerGroupsTest.testDeleteCmdAllGroups(DeleteConsumerGroupsTest.scala:148){quote}
> STDOUT
> {quote}Error: Deletion of some consumer groups failed: * Group 'test.group' 
> could not be deleted due to: java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.GroupNotEmptyException: The group is not 
> empty. Error: Deletion of some consumer groups failed: * Group 
> 'missing.group' could not be deleted due to: 
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.GroupIdNotFoundException: The group id does 
> not exist. [2019-04-16 09:42:02,316] WARN Unable to read additional data from 
> client sessionid 0x104f958dba3, likely client has closed socket 
> (org.apache.zookeeper.server.NIOServerCnxn:376) Deletion of requested 
> consumer groups ('test.group') was successful. Error: Deletion of some 
> consumer groups failed: * Group 'missing.group' could not be deleted due to: 
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.GroupIdNotFoundException: The group id does 
> not exist. These consumer groups were deleted successfully: 
> 'test.group'{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-6793) Unnecessary warning log message

2020-02-17 Thread John Roesler (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038518#comment-17038518
 ] 

John Roesler commented on KAFKA-6793:
-

Amending my previous comment, after looking over the history of this ticket and 
KAFKA-7509, I don't think that it's possible to pass only "known, valid" 
configurations to the clients, since the clients themselves may have injected 
components (like interceptors) for which we can't know the configurations up 
front.

I've sent a message to the dev mailing list "[DISCUSS] KIP-552: Add interface 
to handle unused config" thread with an alternative proposal.

> Unnecessary warning log message 
> 
>
> Key: KAFKA-6793
> URL: https://issues.apache.org/jira/browse/KAFKA-6793
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.1.0
>Reporter: Anna O
>Priority: Minor
>
> When upgraded KafkaStreams from 0.11.0.2 to 1.1.0 the following warning log 
> started to appear:
> level: WARN
> logger: org.apache.kafka.clients.consumer.ConsumerConfig
> message: The configuration 'admin.retries' was supplied but isn't a known 
> config.
> The config is not explicitly supplied to the streams.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (KAFKA-9515) Upgrade ZooKeeper to 3.5.7

2020-02-17 Thread Ismael Juma (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-9515.

Resolution: Fixed

> Upgrade ZooKeeper to 3.5.7
> --
>
> Key: KAFKA-9515
> URL: https://issues.apache.org/jira/browse/KAFKA-9515
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Blocker
> Fix For: 2.5.0, 2.4.1
>
>
> There are some critical fixes in ZK 3.5.7 and the first RC has been posted:
> [https://mail-archives.apache.org/mod_mbox/zookeeper-dev/202002.mbox/%3cCAGH6_KiULzemT-V4x_2ybWeKLMvQ+eh=q-dzsiz8a-ypp5t...@mail.gmail.com%3e]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9515) Upgrade ZooKeeper to 3.5.7

2020-02-17 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038465#comment-17038465
 ] 

ASF GitHub Bot commented on KAFKA-9515:
---

ijuma commented on pull request #8125: KAFKA-9515: Upgrade ZooKeeper to 3.5.7
URL: https://github.com/apache/kafka/pull/8125
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Upgrade ZooKeeper to 3.5.7
> --
>
> Key: KAFKA-9515
> URL: https://issues.apache.org/jira/browse/KAFKA-9515
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Blocker
> Fix For: 2.5.0, 2.4.1
>
>
> There are some critical fixes in ZK 3.5.7 and the first RC has been posted:
> [https://mail-archives.apache.org/mod_mbox/zookeeper-dev/202002.mbox/%3cCAGH6_KiULzemT-V4x_2ybWeKLMvQ+eh=q-dzsiz8a-ypp5t...@mail.gmail.com%3e]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-8967) Flaky test kafka.api.SaslSslAdminIntegrationTest.testCreateTopicsResponseMetadataAndConfig

2020-02-17 Thread Chia-Ping Tsai (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-8967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038402#comment-17038402
 ] 

Chia-Ping Tsai commented on KAFKA-8967:
---

SaslSslAdminClientIntegrationTest was renamed to SaslSslAdminIntegrationTest 
(see 
[this|https://github.com/apache/kafka/commit/400185421f008662ee6f92298154151493486c1e])
 so I updated the title.

> Flaky test 
> kafka.api.SaslSslAdminIntegrationTest.testCreateTopicsResponseMetadataAndConfig
> --
>
> Key: KAFKA-8967
> URL: https://issues.apache.org/jira/browse/KAFKA-8967
> Project: Kafka
>  Issue Type: Test
>  Components: core, security, unit tests
>Reporter: Stanislav Kozlovski
>Priority: Major
>  Labels: flaky-test
>
> {code:java}
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition.
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
>   at 
> kafka.api.SaslSslAdminClientIntegrationTest.testCreateTopicsResponseMetadataAndConfig(SaslSslAdminClientIntegrationTest.scala:452)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: 
> This server does not host this topic-partition.{code}
> Failed in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/25374]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (KAFKA-8268) Flaky Test SaslSslAdminIntegrationTest#testSeekAfterDeleteRecords

2020-02-17 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai updated KAFKA-8268:
--
Summary: Flaky Test SaslSslAdminIntegrationTest#testSeekAfterDeleteRecords  
(was: Flaky Test SaslSslAdminClientIntegrationTest#testSeekAfterDeleteRecords)

> Flaky Test SaslSslAdminIntegrationTest#testSeekAfterDeleteRecords
> -
>
> Key: KAFKA-8268
> URL: https://issues.apache.org/jira/browse/KAFKA-8268
> Project: Kafka
>  Issue Type: Bug
>  Components: core, unit tests
>Affects Versions: 2.3.0
>Reporter: Matthias J. Sax
>Priority: Critical
>  Labels: flaky-test
> Fix For: 2.5.0
>
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3570/tests]
> {quote}java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.TimeoutException: Aborted due to timeout.
> at 
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
>  
> at 
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at 
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at 
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at 
> kafka.api.AdminClientIntegrationTest.testSeekAfterDeleteRecords(AdminClientIntegrationTest.scala:775){quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (KAFKA-8967) Flaky test kafka.api.SaslSslAdminIntegrationTest.testCreateTopicsResponseMetadataAndConfig

2020-02-17 Thread Chia-Ping Tsai (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-8967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai updated KAFKA-8967:
--
Summary: Flaky test 
kafka.api.SaslSslAdminIntegrationTest.testCreateTopicsResponseMetadataAndConfig 
 (was: Flaky test 
kafka.api.SaslSslAdminClientIntegrationTest.testCreateTopicsResponseMetadataAndConfig)

> Flaky test 
> kafka.api.SaslSslAdminIntegrationTest.testCreateTopicsResponseMetadataAndConfig
> --
>
> Key: KAFKA-8967
> URL: https://issues.apache.org/jira/browse/KAFKA-8967
> Project: Kafka
>  Issue Type: Test
>  Components: core, security, unit tests
>Reporter: Stanislav Kozlovski
>Priority: Major
>  Labels: flaky-test
>
> {code:java}
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition.
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
>   at 
> kafka.api.SaslSslAdminClientIntegrationTest.testCreateTopicsResponseMetadataAndConfig(SaslSslAdminClientIntegrationTest.scala:452)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: 
> This server does not host this topic-partition.{code}
> Failed in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/25374]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-8268) Flaky Test SaslSslAdminClientIntegrationTest#testSeekAfterDeleteRecords

2020-02-17 Thread Chia-Ping Tsai (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038401#comment-17038401
 ] 

Chia-Ping Tsai commented on KAFKA-8268:
---

SaslSslAdminClientIntegrationTest was renamed to SaslSslAdminIntegrationTest 
(see 
[this|https://github.com/apache/kafka/commit/400185421f008662ee6f92298154151493486c1e])
 so I updated the title.

> Flaky Test SaslSslAdminClientIntegrationTest#testSeekAfterDeleteRecords
> ---
>
> Key: KAFKA-8268
> URL: https://issues.apache.org/jira/browse/KAFKA-8268
> Project: Kafka
>  Issue Type: Bug
>  Components: core, unit tests
>Affects Versions: 2.3.0
>Reporter: Matthias J. Sax
>Priority: Critical
>  Labels: flaky-test
> Fix For: 2.5.0
>
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3570/tests]
> {quote}java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.TimeoutException: Aborted due to timeout.
> at 
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
>  
> at 
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at 
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at 
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at 
> kafka.api.AdminClientIntegrationTest.testSeekAfterDeleteRecords(AdminClientIntegrationTest.scala:775){quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-6669) KIP-271: Add NetworkClient redirector

2020-02-17 Thread Sergey Korytnikov (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038324#comment-17038324
 ] 

Sergey Korytnikov commented on KAFKA-6669:
--

[~gokul2411s], I didn't find enough interest among Kafka dev list members in 
this proposal to maintain a discussion and proceed with voting. To my 
knowledge, we couldn't find any workaround for Kafka cluster at the time, 
deployed on OpenShift MacOS. This feature has a value for development testing 
mostly and not for production as usually Kafka brokers may not be accessed from 
outside of Kubernetes cluster.

 

> KIP-271: Add NetworkClient redirector
> -
>
> Key: KAFKA-6669
> URL: https://issues.apache.org/jira/browse/KAFKA-6669
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Sergey Korytnikov
>Priority: Major
>
> "KIP-271 Add NetworkClient redirector" purposes the change to accommodate 
> environments without proper DNS support, when KafkaProducer or KafkaConsumer 
> tries to connect to Kafka brokers that are inside another network.
> Currently, Kafka client connection fails with 
> "[java.io|http://java.io/].IOException: Can't resolve address" after Kafka 
> cluster metadata is updated with internal DNS names of brokers, unreachable 
> by Client. The example of such configuration might be a Java Client calling 
> Kafka Cluster from outside of Kubernetes cluster or AWS network.
> The KIP improves NetworkClient to redirect the call to alternative network 
> address and provides developers with additional Configurable interface to 
> specify redirection rules.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-6669) KIP-271: Add NetworkClient redirector

2020-02-17 Thread Gokul Ramanan Subramanian (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038241#comment-17038241
 ] 

Gokul Ramanan Subramanian commented on KAFKA-6669:
--

[~ssbiox] is this still on your radar? Did you find any issues with this 
approach? Did you adopt an alternative approach?

> KIP-271: Add NetworkClient redirector
> -
>
> Key: KAFKA-6669
> URL: https://issues.apache.org/jira/browse/KAFKA-6669
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Sergey Korytnikov
>Priority: Major
>
> "KIP-271 Add NetworkClient redirector" purposes the change to accommodate 
> environments without proper DNS support, when KafkaProducer or KafkaConsumer 
> tries to connect to Kafka brokers that are inside another network.
> Currently, Kafka client connection fails with 
> "[java.io|http://java.io/].IOException: Can't resolve address" after Kafka 
> cluster metadata is updated with internal DNS names of brokers, unreachable 
> by Client. The example of such configuration might be a Java Client calling 
> Kafka Cluster from outside of Kubernetes cluster or AWS network.
> The KIP improves NetworkClient to redirect the call to alternative network 
> address and provides developers with additional Configurable interface to 
> specify redirection rules.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9319) Run some system tests using TLSv1.3

2020-02-17 Thread Rajini Sivaram (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038220#comment-17038220
 ] 

Rajini Sivaram commented on KAFKA-9319:
---

Can you rerun the tests that failed on their own and see if those are 
intermittent failures? If they continue to fail, it will be good to check if 
they pass with TLSv1.2.

> Run some system tests using TLSv1.3
> ---
>
> Key: KAFKA-9319
> URL: https://issues.apache.org/jira/browse/KAFKA-9319
> Project: Kafka
>  Issue Type: Test
>Reporter: Rajini Sivaram
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.5.0
>
>
> KAFKA-7251 enables TLSv1.3 for Kafka. We should get some system tests to run 
> using TLSv1.3. Since TLSv1.3 is only supported from Java 11 onwards, we need 
> a system test build that runs with JDK 11 to enable these tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-6817) UnknownProducerIdException when writing messages with old timestamps

2020-02-17 Thread chris (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038219#comment-17038219
 ] 

chris commented on KAFKA-6817:
--

I've also encountered this problem. We are reading 4 month old data, using 
Kafka 2.3.0. As a workaround i've changed the processing guarantee from 
exactly_one to at_least_once, to avoid using the transactional produce. In this 
case using at_least_once is acceptable, but a proper fix would be great.

> UnknownProducerIdException when writing messages with old timestamps
> 
>
> Key: KAFKA-6817
> URL: https://issues.apache.org/jira/browse/KAFKA-6817
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 1.1.0
>Reporter: Odin Standal
>Priority: Major
>
> We are seeing the following exception in our Kafka application: 
> {code:java}
> ERROR o.a.k.s.p.internals.StreamTask - task [0_0] Failed to close producer 
> due to the following error: org.apache.kafka.streams.errors.StreamsException: 
> task [0_0] Abort sending since an error caught with a previous record (key 
> 22 value some-value timestamp 1519200902670) to topic 
> exactly-once-test-topic- v2 due to This exception is raised by the broker if 
> it could not locate the producer metadata associated with the producerId in 
> question. This could happen if, for instance, the producer's records were 
> deleted because their retention time had elapsed. Once the last records of 
> the producerId are removed, the producer's metadata is removed from the 
> broker, and future appends by the producer will return this exception. at 
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:125)
>  at 
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.access$500(RecordCollectorImpl.java:48)
>  at 
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl$1.onCompletion(RecordCollectorImpl.java:180)
>  at 
> org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1199)
>  at 
> org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:204)
>  at 
> org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:187)
>  at 
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:627) 
> at 
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:596) 
> at 
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:557)
>  at 
> org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:481)
>  at 
> org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74) 
> at 
> org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:692)
>  at 
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101) 
> at 
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:482)
>  at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:474) at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) at 
> java.lang.Thread.run(Thread.java:748) Caused by: 
> org.apache.kafka.common.errors.UnknownProducerIdException
> {code}
> We discovered this error when we had the need to reprocess old messages. See 
> more details on 
> [Stackoverflow|https://stackoverflow.com/questions/49872827/unknownproduceridexception-in-kafka-streams-when-enabling-exactly-once?noredirect=1#comment86901917_49872827]
> We have reproduced the error with a smaller example application. The error 
> occurs after 10 minutes of producing messages that have old timestamps (type 
> 1 year old). The topic we are writing to has a retention.ms set to 1 year so 
> we are expecting the messages to stay there.
> After digging through the ProducerStateManager-code in the Kafka source code 
> we have a theory of what might be wrong.
> The ProducerStateManager.removeExpiredProducers() seems to remove producers 
> from memory erroneously when processing records which are older than the 
> maxProducerIdExpirationMs (coming from the `transactional.id.expiration.ms` 
> configuration), which is set by default to 7 days. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9319) Run some system tests using TLSv1.3

2020-02-17 Thread Nikolay Izhikov (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038214#comment-17038214
 ] 

Nikolay Izhikov commented on KAFKA-9319:


Hello, [~rsivaram].

I think we are ready to proceed with the KAFKA-9320 and enable TLSv1.3 by 
default.
What do you think?

> Run some system tests using TLSv1.3
> ---
>
> Key: KAFKA-9319
> URL: https://issues.apache.org/jira/browse/KAFKA-9319
> Project: Kafka
>  Issue Type: Test
>Reporter: Rajini Sivaram
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.5.0
>
>
> KAFKA-7251 enables TLSv1.3 for Kafka. We should get some system tests to run 
> using TLSv1.3. Since TLSv1.3 is only supported from Java 11 onwards, we need 
> a system test build that runs with JDK 11 to enable these tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-8835) Update documentation for URP changes in KIP-352

2020-02-17 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-8835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038209#comment-17038209
 ] 

ASF GitHub Bot commented on KAFKA-8835:
---

viktorsomogyi commented on pull request #8127: KAFKA-8835: KIP-352 docs update
URL: https://github.com/apache/kafka/pull/8127
 
 
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update documentation for URP changes in KIP-352
> ---
>
> Key: KAFKA-8835
> URL: https://issues.apache.org/jira/browse/KAFKA-8835
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Viktor Somogyi-Vass
>Priority: Major
>
> This Jira covers any doc changes needed for the changes to URP semantics in 
> KIP-352: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9319) Run some system tests using TLSv1.3

2020-02-17 Thread ASF GitHub Bot (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038203#comment-17038203
 ] 

ASF GitHub Bot commented on KAFKA-9319:
---

rajinisivaram commented on pull request #8106: KAFKA-9319: Fix generation of CA 
certificate for system tests.
URL: https://github.com/apache/kafka/pull/8106
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Run some system tests using TLSv1.3
> ---
>
> Key: KAFKA-9319
> URL: https://issues.apache.org/jira/browse/KAFKA-9319
> Project: Kafka
>  Issue Type: Test
>Reporter: Rajini Sivaram
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.5.0
>
>
> KAFKA-7251 enables TLSv1.3 for Kafka. We should get some system tests to run 
> using TLSv1.3. Since TLSv1.3 is only supported from Java 11 onwards, we need 
> a system test build that runs with JDK 11 to enable these tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)