[jira] [Commented] (KAFKA-13606) MirrorCheckpointTask doesn't check offsets sync result

2022-01-22 Thread Kvicii.Yu (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480369#comment-17480369
 ] 

Kvicii.Yu commented on KAFKA-13606:
---

hi, [~savulchik] Can you elaborate on how this problem needs to be optimized?

> MirrorCheckpointTask doesn't check offsets sync result
> --
>
> Key: KAFKA-13606
> URL: https://issues.apache.org/jira/browse/KAFKA-13606
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.7.0, 2.8.0, 2.7.1, 2.7.2, 2.8.1, 3.0.0
>Reporter: Stanislav Savulchik
>Priority: Major
>
> {{MirrorCheckpointTask}} doesn't check the result of calling 
> {{AdminClient#alterConsumerGroupOffsets}} method that could return a failed 
> Future for instance due to incorrect ACL in the target kafka cluster.
> [https://github.com/apache/kafka/blob/3.0.0/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointTask.java#L302]
> I guess it should at least log or even rethrow the exception.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Created] (KAFKA-13612) internal topics won't be created in metadataRequest when auto.create.topics.enable=false

2022-01-22 Thread Luke Chen (Jira)
Luke Chen created KAFKA-13612:
-

 Summary: internal topics won't be created in metadataRequest when 
auto.create.topics.enable=false
 Key: KAFKA-13612
 URL: https://issues.apache.org/jira/browse/KAFKA-13612
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.0.0, 2.8.0
Reporter: Luke Chen
Assignee: Luke Chen


In KAFKA-9751, when create internal topics through FindCoordinator or Metadata 
request, we route the topic creation request to the controller instead of 
handling by itself. We change logic in `KafkaApis#getTopicMetadata`, and make 
the internal topic won't get created when "auto.create.topics.enable=false`. 
h4.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Updated] (KAFKA-13607) Cannot use PEM certificate coding when parent defined file-based

2022-01-22 Thread Piotr Smolinski (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Smolinski updated KAFKA-13607:

Description: 
The problem applies to the situation when we create a Kafka client based on 
prepopulated config. If we have only partial control on the input we can 
attempt to reset some values.

KIP-651 added a new cool feature to use PEM coding of certificates as an 
alternative to file stores. I have observed a problem in Confluent Replicator. 
We have shifted the common configuration to the worker level and assumed the 
connectors define only what is specific for them. The security setup is mTLS, 
i.e. we need both client cert and trusted chain. Our default configuration has 
both in #PKCS12 files, but we had to reverse the replication direction and 
redefine the destination coordinates. For these we have certificates, but 
having KIP-651 we could specify them as connector params as opposed to the 
worker deployment change.

It came out that we cannot override {*}ssl.keystore.location{*}, 
{*}ssl.keystore.password{*}, etc. simply with empty values, because the code in 
the *DefaultSslEngineFactory* checks if the entry is null. We can only override 
it to empty string.

*DefaultSslEngineFactory* should treat the unexpected configuration entries as 
absent when they are {*}null{*}, but also when the given entry is an empty 
string.

For a workaround I have created a hacky patch that fixes the behaviour:

[https://github.com/piotrsmolinski/kafka-ssl-fix]

 

  was:
The problem applies to the situation when we create a Kafka client based on 
prepopulated config. If we have only partial control on the input we can 
attempt to reset some values.

KIP-651 added a new cool feature to use PEM coding of certificates as an 
alternative to file stores. I have observed a problem in Confluent Replicator. 
We have shifted the common configuration to the worker level and assumed the 
connectors define only what is specific for them. The security setup is mTLS, 
i.e. we need both client cert and trusted chain. Our default configuration has 
both in #PKCS12 files, but we had to reverse the replication direction and 
redefine the destination coordinates. For these we have certificates, but 
having KIP-651 we could specify them as connector params as opposed to the 
worker deployment change.

It came out that we cannot override **ssl.keystore.location**, 
**ssl.keystore.password**, etc. simply with empty values, because the code in 
the **DefaultSslEngineFactory** checks if the entry is null. We can only 
override it to empty string.

**DefaultSslEngineFactory** should treat the unexpected configuration entries 
as absent when they are **null**, but also when the given entry is an empty 
string.

For a workaround I have created a hacky patch that fixes the behaviour:

https://github.com/piotrsmolinski/kafka-ssl-fix

 


> Cannot use PEM certificate coding when parent defined file-based
> 
>
> Key: KAFKA-13607
> URL: https://issues.apache.org/jira/browse/KAFKA-13607
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, config, KafkaConnect
>Affects Versions: 2.7.1, 3.0.0
>Reporter: Piotr Smolinski
>Priority: Major
>
> The problem applies to the situation when we create a Kafka client based on 
> prepopulated config. If we have only partial control on the input we can 
> attempt to reset some values.
> KIP-651 added a new cool feature to use PEM coding of certificates as an 
> alternative to file stores. I have observed a problem in Confluent 
> Replicator. We have shifted the common configuration to the worker level and 
> assumed the connectors define only what is specific for them. The security 
> setup is mTLS, i.e. we need both client cert and trusted chain. Our default 
> configuration has both in #PKCS12 files, but we had to reverse the 
> replication direction and redefine the destination coordinates. For these we 
> have certificates, but having KIP-651 we could specify them as connector 
> params as opposed to the worker deployment change.
> It came out that we cannot override {*}ssl.keystore.location{*}, 
> {*}ssl.keystore.password{*}, etc. simply with empty values, because the code 
> in the *DefaultSslEngineFactory* checks if the entry is null. We can only 
> override it to empty string.
> *DefaultSslEngineFactory* should treat the unexpected configuration entries 
> as absent when they are {*}null{*}, but also when the given entry is an empty 
> string.
> For a workaround I have created a hacky patch that fixes the behaviour:
> [https://github.com/piotrsmolinski/kafka-ssl-fix]
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[GitHub] [kafka] sayantanu-dey opened a new pull request #11703: KAFKA-13588: consolidate `changelogFor` methods to simplify the generation of internal topic names

2022-01-22 Thread GitBox


sayantanu-dey opened a new pull request #11703:
URL: https://github.com/apache/kafka/pull/11703


   *More detailed description of your change,
   if necessary. The PR title and PR message become
   the squashed commit message, so use a separate
   comment to ping reviewers.*
   
   *Summary of testing strategy (including rationale)
   for the feature or bug fix. Unit and/or integration
   tests are expected for any behaviour change and
   system tests should be considered for larger changes.*
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Commented] (KAFKA-13588) We should consolidate `changelogFor` methods to simplify the generation of internal topic names

2022-01-22 Thread Sayantanu Dey (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480412#comment-17480412
 ] 

Sayantanu Dey commented on KAFKA-13588:
---

[~guozhang] [~wcarlson5]  Hey, I have raised a pull request to fix this issue.

PR: https://github.com/apache/kafka/pull/11703

> We should consolidate `changelogFor` methods to simplify the generation of 
> internal topic names
> ---
>
> Key: KAFKA-13588
> URL: https://issues.apache.org/jira/browse/KAFKA-13588
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Walker Carlson
>Assignee: Sayantanu Dey
>Priority: Minor
>  Labels: newbie
> Fix For: 3.3.0
>
>
> [https://github.com/apache/kafka/pull/11611#discussion_r772625486]
> we should use `ProcessorContextUtils#changelogFor` after we remove 
> `init(final ProcessorContext context, final StateStore root)` in 
> `CahceingWindowStore#initInternal` --- this will happen in around Dec.2022, 
> which is around 3.3.0
>  
> Or any other place that we generate an internal topic name.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] (KAFKA-13588) We should consolidate `changelogFor` methods to simplify the generation of internal topic names

2022-01-22 Thread Sayantanu Dey (Jira)


[ https://issues.apache.org/jira/browse/KAFKA-13588 ]


Sayantanu Dey deleted comment on KAFKA-13588:
---

was (Author: JIRAUSER283615):
PR: https://github.com/apache/kafka/pull/11703

> We should consolidate `changelogFor` methods to simplify the generation of 
> internal topic names
> ---
>
> Key: KAFKA-13588
> URL: https://issues.apache.org/jira/browse/KAFKA-13588
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Walker Carlson
>Assignee: Sayantanu Dey
>Priority: Minor
>  Labels: newbie
> Fix For: 3.3.0
>
>
> [https://github.com/apache/kafka/pull/11611#discussion_r772625486]
> we should use `ProcessorContextUtils#changelogFor` after we remove 
> `init(final ProcessorContext context, final StateStore root)` in 
> `CahceingWindowStore#initInternal` --- this will happen in around Dec.2022, 
> which is around 3.3.0
>  
> Or any other place that we generate an internal topic name.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Commented] (KAFKA-13322) Java client produces a large amount of garbage during a poll

2022-01-22 Thread Ismael Juma (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-13322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480458#comment-17480458
 ] 

Ismael Juma commented on KAFKA-13322:
-

Thanks for the ticket. To be clear, this issue is not specific to 3.0.0, right?

> Java client produces a large amount of garbage during a poll
> 
>
> Key: KAFKA-13322
> URL: https://issues.apache.org/jira/browse/KAFKA-13322
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 3.0.0
>Reporter: Michail
>Priority: Minor
>
> The java kafka consumer creates multiple collections during a single poll 
> command: in my test system i have a consumer that polls a topic with 100 
> partitions and even though no messages are coming through, the code allocates 
> around 100M per 5 minutes.
>  
> I've investigated the allocations and the biggest ones can be easily avoided 
> by moving them to the instance level, something that can be done as 
> KafkaConsumer is not thread safe. Purpose of this Jira is to get rid of most 
> of them applying either this or a similar approach.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] (KAFKA-13588) We should consolidate `changelogFor` methods to simplify the generation of internal topic names

2022-01-22 Thread Sayantanu Dey (Jira)


[ https://issues.apache.org/jira/browse/KAFKA-13588 ]


Sayantanu Dey deleted comment on KAFKA-13588:
---

was (Author: JIRAUSER283615):
[~guozhang] [~wcarlson5]  Hey, I have raised a pull request to fix this issue.

PR: https://github.com/apache/kafka/pull/11703

> We should consolidate `changelogFor` methods to simplify the generation of 
> internal topic names
> ---
>
> Key: KAFKA-13588
> URL: https://issues.apache.org/jira/browse/KAFKA-13588
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Walker Carlson
>Assignee: Sayantanu Dey
>Priority: Minor
>  Labels: newbie
> Fix For: 3.3.0
>
>
> [https://github.com/apache/kafka/pull/11611#discussion_r772625486]
> we should use `ProcessorContextUtils#changelogFor` after we remove 
> `init(final ProcessorContext context, final StateStore root)` in 
> `CahceingWindowStore#initInternal` --- this will happen in around Dec.2022, 
> which is around 3.3.0
>  
> Or any other place that we generate an internal topic name.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Commented] (KAFKA-13606) MirrorCheckpointTask doesn't check offsets sync result

2022-01-22 Thread Stanislav Savulchik (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480461#comment-17480461
 ] 

Stanislav Savulchik commented on KAFKA-13606:
-

HI, [~Kvicii].

I propose to just log the exception of a returned failed Future in order to 
make it visible in logs because right now we have no traces of the problem.

Re-throwing the exception seems a bad option to me because a task could sync 
offsets of many consumer groups and we shouldn't prevent other consumer groups 
from syncing despite of the occurred exception.

> MirrorCheckpointTask doesn't check offsets sync result
> --
>
> Key: KAFKA-13606
> URL: https://issues.apache.org/jira/browse/KAFKA-13606
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.7.0, 2.8.0, 2.7.1, 2.7.2, 2.8.1, 3.0.0
>Reporter: Stanislav Savulchik
>Priority: Major
>
> {{MirrorCheckpointTask}} doesn't check the result of calling 
> {{AdminClient#alterConsumerGroupOffsets}} method that could return a failed 
> Future for instance due to incorrect ACL in the target kafka cluster.
> [https://github.com/apache/kafka/blob/3.0.0/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointTask.java#L302]
> I guess it should at least log or even rethrow the exception.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Comment Edited] (KAFKA-13606) MirrorCheckpointTask doesn't check offsets sync result

2022-01-22 Thread Stanislav Savulchik (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480461#comment-17480461
 ] 

Stanislav Savulchik edited comment on KAFKA-13606 at 1/22/22, 3:16 PM:
---

Hi, [~Kvicii].

I propose to just log the exception of a returned failed Future in order to 
make it visible in logs because right now we have no traces of the problem.

Re-throwing the exception seems a bad option to me because a task could sync 
offsets of many consumer groups and we shouldn't prevent other consumer groups 
from syncing despite of the occurred exception.


was (Author: savulchik):
HI, [~Kvicii].

I propose to just log the exception of a returned failed Future in order to 
make it visible in logs because right now we have no traces of the problem.

Re-throwing the exception seems a bad option to me because a task could sync 
offsets of many consumer groups and we shouldn't prevent other consumer groups 
from syncing despite of the occurred exception.

> MirrorCheckpointTask doesn't check offsets sync result
> --
>
> Key: KAFKA-13606
> URL: https://issues.apache.org/jira/browse/KAFKA-13606
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.7.0, 2.8.0, 2.7.1, 2.7.2, 2.8.1, 3.0.0
>Reporter: Stanislav Savulchik
>Priority: Major
>
> {{MirrorCheckpointTask}} doesn't check the result of calling 
> {{AdminClient#alterConsumerGroupOffsets}} method that could return a failed 
> Future for instance due to incorrect ACL in the target kafka cluster.
> [https://github.com/apache/kafka/blob/3.0.0/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointTask.java#L302]
> I guess it should at least log or even rethrow the exception.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Assigned] (KAFKA-13602) Allow to broadcast a result record

2022-01-22 Thread Florin Akermann (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Florin Akermann reassigned KAFKA-13602:
---

Assignee: Florin Akermann

> Allow to broadcast a result record
> --
>
> Key: KAFKA-13602
> URL: https://issues.apache.org/jira/browse/KAFKA-13602
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Florin Akermann
>Priority: Major
>  Labels: needs-kip, newbie++
>
> From time to time, users ask how they can send a record to more than one 
> partition in a sink topic. Currently, this is only possible by replicate the 
> message N times before the sink and use a custom partitioner to write the N 
> messages into the N different partitions.
> It might be worth to make this easier and add a new feature for it. There are 
> multiple options:
>  * extend `to()` / `addSink()` with a "broadcast" option/config
>  * add `toAllPartitions()` / `addBroadcastSink()` methods
>  * allow StreamPartitioner to return `-1` for "all partitions"
>  * extend `StreamPartitioner` to allow returning more than one partition (ie 
> a list/collection of integers instead of a single int)
> The first three options imply that a "full broadcast" is supported only, so 
> it's less flexible. On the other hand, it's easier to use (especially the 
> first two options are easy as they do not require to implement a custom 
> partitioner).
> The last option would be most flexible and also allow for a "partial 
> broadcast" (aka multi-cast) pattern. It might also be possible to combine two 
> options, or maye even a totally different idea.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Commented] (KAFKA-13606) MirrorCheckpointTask doesn't check offsets sync result

2022-01-22 Thread Kvicii.Yu (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480490#comment-17480490
 ] 

Kvicii.Yu commented on KAFKA-13606:
---

[~savulchik] 

Therefore, my understanding is that we should call partitionResult in the 
method syncGroupOffset to handle whether an exception occurs, and record the 
log if an exception occurs.

> MirrorCheckpointTask doesn't check offsets sync result
> --
>
> Key: KAFKA-13606
> URL: https://issues.apache.org/jira/browse/KAFKA-13606
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.7.0, 2.8.0, 2.7.1, 2.7.2, 2.8.1, 3.0.0
>Reporter: Stanislav Savulchik
>Priority: Major
>
> {{MirrorCheckpointTask}} doesn't check the result of calling 
> {{AdminClient#alterConsumerGroupOffsets}} method that could return a failed 
> Future for instance due to incorrect ACL in the target kafka cluster.
> [https://github.com/apache/kafka/blob/3.0.0/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointTask.java#L302]
> I guess it should at least log or even rethrow the exception.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Updated] (KAFKA-13322) Java client produces a large amount of garbage during a poll

2022-01-22 Thread Guozhang Wang (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-13322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-13322:
--
Labels: new-rebalance-should-fix  (was: )

> Java client produces a large amount of garbage during a poll
> 
>
> Key: KAFKA-13322
> URL: https://issues.apache.org/jira/browse/KAFKA-13322
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 3.0.0
>Reporter: Michail
>Priority: Minor
>  Labels: new-rebalance-should-fix
>
> The java kafka consumer creates multiple collections during a single poll 
> command: in my test system i have a consumer that polls a topic with 100 
> partitions and even though no messages are coming through, the code allocates 
> around 100M per 5 minutes.
>  
> I've investigated the allocations and the biggest ones can be easily avoided 
> by moving them to the instance level, something that can be done as 
> KafkaConsumer is not thread safe. Purpose of this Jira is to get rid of most 
> of them applying either this or a similar approach.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Assigned] (KAFKA-13602) Allow to broadcast a result record

2022-01-22 Thread Florin Akermann (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Florin Akermann reassigned KAFKA-13602:
---

Assignee: (was: Florin Akermann)

> Allow to broadcast a result record
> --
>
> Key: KAFKA-13602
> URL: https://issues.apache.org/jira/browse/KAFKA-13602
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Matthias J. Sax
>Priority: Major
>  Labels: needs-kip, newbie++
>
> From time to time, users ask how they can send a record to more than one 
> partition in a sink topic. Currently, this is only possible by replicate the 
> message N times before the sink and use a custom partitioner to write the N 
> messages into the N different partitions.
> It might be worth to make this easier and add a new feature for it. There are 
> multiple options:
>  * extend `to()` / `addSink()` with a "broadcast" option/config
>  * add `toAllPartitions()` / `addBroadcastSink()` methods
>  * allow StreamPartitioner to return `-1` for "all partitions"
>  * extend `StreamPartitioner` to allow returning more than one partition (ie 
> a list/collection of integers instead of a single int)
> The first three options imply that a "full broadcast" is supported only, so 
> it's less flexible. On the other hand, it's easier to use (especially the 
> first two options are easy as they do not require to implement a custom 
> partitioner).
> The last option would be most flexible and also allow for a "partial 
> broadcast" (aka multi-cast) pattern. It might also be possible to combine two 
> options, or maye even a totally different idea.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[GitHub] [kafka] vamossagar12 commented on pull request #11424: KAFKA-13152: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

2022-01-22 Thread GitBox


vamossagar12 commented on pull request #11424:
URL: https://github.com/apache/kafka/pull/11424#issuecomment-1019400642


   @guozhangwang , i fixed that. Hopefully this time we won't see errors 
related to this PR. I will also check it. Also, @ableegoldman , thanks for 
confirming the metric name. I have updated the PR and also sent out an update 
on the KIP.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [kafka] showuon closed pull request #11658: [WIP] MINOR: test leaderElection

2022-01-22 Thread GitBox


showuon closed pull request #11658:
URL: https://github.com/apache/kafka/pull/11658


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Commented] (KAFKA-13349) Allow Iterator.remove on KeyValueIterator

2022-01-22 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480539#comment-17480539
 ] 

Sagar Rao commented on KAFKA-13349:
---

Thanks [~guozhang] !

> Allow Iterator.remove on KeyValueIterator
> -
>
> Key: KAFKA-13349
> URL: https://issues.apache.org/jira/browse/KAFKA-13349
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Sagar Rao
>Priority: Major
>  Labels: needs-kip, newbie++
>
> Today Stream's state store's range iterator does not support `remove`. We 
> could consider adding such support for all the built-in state stores:
> * RocksDB's native iterator does not support removal, but we can always do a 
> delete(key) concurrently while the iterator is open on the snapshot.
> * In-Memory: straight forward implementation.
> The benefit of that is then for range-and-delete truncation operation we do 
> not necessarily have to be cautious about concurrent modification exceptions. 
> This could also help GC with in-memory stores.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Comment Edited] (KAFKA-13588) We should consolidate `changelogFor` methods to simplify the generation of internal topic names

2022-01-22 Thread Sayantanu Dey (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480498#comment-17480498
 ] 

Sayantanu Dey edited comment on KAFKA-13588 at 1/23/22, 5:49 AM:
-

pr to fix the issue: https://github.com/apache/kafka/pull/11703

1 test case got failed on Jenkins but on local, the failed test is passing.


was (Author: JIRAUSER283615):
pr to fix the issue: 
[https://github.com/apache/kafka/pull/11703|https://github.com/apache/kafka/pull/11703,]

1 test case got failed on Jenkins but on local, the failed test is passing.

> We should consolidate `changelogFor` methods to simplify the generation of 
> internal topic names
> ---
>
> Key: KAFKA-13588
> URL: https://issues.apache.org/jira/browse/KAFKA-13588
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Walker Carlson
>Assignee: Sayantanu Dey
>Priority: Minor
>  Labels: newbie
> Fix For: 3.3.0
>
>
> [https://github.com/apache/kafka/pull/11611#discussion_r772625486]
> we should use `ProcessorContextUtils#changelogFor` after we remove 
> `init(final ProcessorContext context, final StateStore root)` in 
> `CahceingWindowStore#initInternal` --- this will happen in around Dec.2022, 
> which is around 3.3.0
>  
> Or any other place that we generate an internal topic name.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[GitHub] [kafka] sayantanu-dey opened a new pull request #11704: KAFKA-13590: rename InternalTopologyBuilder#topicGroups

2022-01-22 Thread GitBox


sayantanu-dey opened a new pull request #11704:
URL: https://github.com/apache/kafka/pull/11704


   *More detailed description of your change,
   if necessary. The PR title and PR message become
   the squashed commit message, so use a separate
   comment to ping reviewers.*
   
   *Summary of testing strategy (including rationale)
   for the feature or bug fix. Unit and/or integration
   tests are expected for any behaviour change and
   system tests should be considered for larger changes.*
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[jira] [Assigned] (KAFKA-13590) Rename InternalTopologyBuilder's #topicGroups method to be more descriptive

2022-01-22 Thread Sayantanu Dey (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sayantanu Dey reassigned KAFKA-13590:
-

Assignee: Sayantanu Dey

> Rename InternalTopologyBuilder's #topicGroups method to be more descriptive
> ---
>
> Key: KAFKA-13590
> URL: https://issues.apache.org/jira/browse/KAFKA-13590
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Assignee: Sayantanu Dey
>Priority: Minor
>  Labels: newbie
>
> Pretty much what the title says, it can be difficult to figure out what this 
> method is actually returning based on the method name.. At least javadocs 
> were added in a recent PR, but  ideally you nwouldn't need to visit the 
> method's implementation at all to understand its function
>  
> See 
> [https://github.com/apache/kafka/pull/11600/files#r768947553|https://github.com/apache/kafka/pull/11600/files#r768947553]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Commented] (KAFKA-13590) Rename InternalTopologyBuilder's #topicGroups method to be more descriptive

2022-01-22 Thread Sayantanu Dey (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480564#comment-17480564
 ] 

Sayantanu Dey commented on KAFKA-13590:
---

pr to fix this issue: https://github.com/apache/kafka/pull/11704

> Rename InternalTopologyBuilder's #topicGroups method to be more descriptive
> ---
>
> Key: KAFKA-13590
> URL: https://issues.apache.org/jira/browse/KAFKA-13590
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: A. Sophie Blee-Goldman
>Assignee: Sayantanu Dey
>Priority: Minor
>  Labels: newbie
>
> Pretty much what the title says, it can be difficult to figure out what this 
> method is actually returning based on the method name.. At least javadocs 
> were added in a recent PR, but  ideally you nwouldn't need to visit the 
> method's implementation at all to understand its function
>  
> See 
> [https://github.com/apache/kafka/pull/11600/files#r768947553|https://github.com/apache/kafka/pull/11600/files#r768947553]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)