[jira] [Created] (KAFKA-15620) Duplicate remote log DELETE_SEGMENT metadata is generated when there are multiple leader epochs in the segment

Henry Cai created KAFKA-15620: - Summary: Duplicate remote log DELETE_SEGMENT metadata is generated when there are multiple leader epochs in the segment Key: KAFKA-15620 URL: https://issues.apache.org/jira/browse

Re: [DISCUSS] KIP-712: Shallow Mirroring

oducer > could be factored into an abstract class. > } > ``` > > Really I'm just re-stating Jason's KIP-706 idea in the context of this KIP, > but it would address the type safety issue and also enable a batch consumer > to have its own set of configs. It also all

Re: [DISCUSS] KIP-712: Shallow Mirroring

a topic whether it has EOS data or not. So, there is no easy > way to skip EOS data from the source. > > Jun > > On Thu, Apr 1, 2021 at 1:07 AM Henry Cai > wrote: > > > Jun, > > > > Thanks for your insight looking into this KIP, we do believe the shallow >

Re: [DISCUSS] KIP-712: Shallow Mirroring

ion in the produce request. This seems not > strictly required in KIP-98. However, changing this will probably add a bit > more complexity in the producer. So, it would be useful to understand its > benefits, especially since it doesn't seem to directly help reduce the CPU > cost in MM

Re: [DISCUSS] KIP-712: Shallow Mirroring

t > > have a number of special cases. The more special cases you have, the > > tougher it becomes to test all the edge cases. > > > > Ismael > > > > On Mon, Mar 29, 2021 at 9:51 PM Henry Cai > > wrote: > > > > > It's interesting th

Re: [DISCUSS] KIP-712: Shallow Mirroring

e want the protocol to be generic and not > have a number of special cases. The more special cases you have, the > tougher it becomes to test all the edge cases. > > Ismael > > On Mon, Mar 29, 2021 at 9:51 PM Henry Cai > wrote: > > > It's interesting this VOTE

Re: [DISCUSS] KIP-712: Shallow Mirroring

> > > > > > Retitled the thread to conform to the common format. > > > > > > > > > > On Fri, Feb 5, 2021 at 4:00 PM Ning Zhang > > > > wrote: > > > > > > > > > > > Hello Henry, > > > > > > > > &

Re: [VOTE] KIP-712: Shallow Mirroring

ard to this. > > But N.B. it looks like KIP-720 will pass, which deprecates MM1. I don't > think there is any reason both KIPs can't pass, but it looks like any > classes introduced in KIP-712 would get immediately deprecated. > > Ryanne > > On Thu, Mar 25, 2021,

[VOTE] KIP-712: Shallow Mirroring

Hi, I'd like to start a vote on KIP-712: Shallow Mirroring. https://cwiki.apache.org/confluence/display/KAFKA/KIP-712%3A+Shallow+Mirroring Thanks

Re: [DISCUSS] KIP-712: Shallow Mirroring

at scale. >> >> I'm bumping up this thread in case there are additional feedback or >> comments. >> >> Thanks, >> --Vahid >> >> On Sat, Feb 13, 2021, 13:59 Ryanne Dolan wrote: >> >> > Glad to hear that latency and thruput aren't

Re: [DISCUSS] KIP-712: Shallow Mirroring

lysis based on these changes. Looking > forward to a PR! > > Ryanne > > On Wed, Feb 10, 2021, 3:50 PM Henry Cai wrote: > >> On the question "whether shallow mirror is only applied on mirror maker >> v1", the code change is mostly on consumer and producer co

Re: [DISCUSS] KIP-712: Shallow Mirroring

sing data in mirror maker. > > > > Probably one thing may need to clarify is: how "shallow" mirroring is > only > > applied to mirrormaker use case, if the changes need to be made on > generic > > consumer and producer (e.g. by adding `fetch.raw.bytes` and >

[jira] [Created] (KAFKA-12295) Shallow Mirroring

Henry Cai created KAFKA-12295: - Summary: Shallow Mirroring Key: KAFKA-12295 URL: https://issues.apache.org/jira/browse/KAFKA-12295 Project: Kafka Issue Type: Improvement Components

Proposing KIP-712: Shallow Mirroring

Dear Community members, We are proposing a new feature to improve the performance of Kafka mirror maker: https://cwiki.apache.org/confluence/display/KAFKA/KIP-712%3A+Shallow+Mirroring The current Kafka MirrorMaker process (with the underlying Consumer and Producer library) uses significant CPU cy

Need permission to create a KIP

Hi, Can someone give me permission to create a KIP? My Apache Id is hcai ( h...@pinterest.com) https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals Thanks.

[jira] [Created] (KAFKA-8089) High level consumer from MirrorMaker is slow to deal with SSL certification expiration

Henry Cai created KAFKA-8089: Summary: High level consumer from MirrorMaker is slow to deal with SSL certification expiration Key: KAFKA-8089 URL: https://issues.apache.org/jira/browse/KAFKA-8089 Project

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

transaction can also rolled back and re-apply during kafka transaction boundary? On Thu, Dec 1, 2016 at 11:05 AM, Guozhang Wang wrote: > @Henry Cai, > > I am working on a separate KIP on Streams to leverage this KIP to have > exactly-once processing semantics (note the exactly-once pro

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

During the 'consumer-transform-produce' cycle, if the consumer app needs to update an external data store (e.g. update RocksDB in Kafka streams), and that update is not idempotent (e.g. increment a counter in RocksDB), how do you make that update part of Kafka transaction? On Wed, Nov 30, 2016 at

Re: [VOTE] 0.10.1.0 RC0

Why is this feature in the release note? - [KAFKA-264 ] - Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator I thought this was already done in 2015. On Thu, Oct 6, 2016 at 4:55 PM, Vahid

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

r a listener, they > provide > > a host:port in config. It is expected they will start a service running > on > > that host:port that they can use to connect to the running KafkaStreams > > Instance. > > > > Thanks, > > Damian > > > > On Thu, 7

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

It wasn't quite clear to me how the user program interacts with the discovery API, especially on the user supplied listener part, how does the user program supply that listener to KafkaStreams and how does KafkaStreams know which port the user listener is running, maybe a more complete end-to-end e

[jira] [Resolved] (KAFKA-3904) File descriptor leaking (Too many open files) for long running stream process

[ https://issues.apache.org/jira/browse/KAFKA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Cai resolved KAFKA-3904. -- Resolution: Duplicate Dup of KAFKA-3619 > File descriptor leaking (Too many open files) for l

[jira] [Commented] (KAFKA-3904) File descriptor leaking (Too many open files) for long running stream process

[ https://issues.apache.org/jira/browse/KAFKA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358038#comment-15358038 ] Henry Cai commented on KAFKA-3904: -- This is a dup of KAFKA-3619 > File des

[jira] [Commented] (KAFKA-3904) File descriptor leaking (Too many open files) for long running stream process

[ https://issues.apache.org/jira/browse/KAFKA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352387#comment-15352387 ] Henry Cai commented on KAFKA-3904: -- Please take a look at this PR for the fix: h

[jira] [Commented] (KAFKA-3904) File descriptor leaking (Too many open files) for long running stream process

[ https://issues.apache.org/jira/browse/KAFKA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352126#comment-15352126 ] Henry Cai commented on KAFKA-3904: -- I took a look at FileChannel.open, looks lik

[jira] [Commented] (KAFKA-3904) File descriptor leaking (Too many open files) for long running stream process

[ https://issues.apache.org/jira/browse/KAFKA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350018#comment-15350018 ] Henry Cai commented on KAFKA-3904: -- I have the fix also: -FileChannel cha

[jira] [Updated] (KAFKA-3904) File descriptor leaking (Too many open files) for long running stream process

[ https://issues.apache.org/jira/browse/KAFKA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Cai updated KAFKA-3904: - Description: I noticed when my application was running long (> 1 day), I will get 'Too many op

[jira] [Created] (KAFKA-3904) File descriptor leaking (Too many open files) for long running stream process

Henry Cai created KAFKA-3904: Summary: File descriptor leaking (Too many open files) for long running stream process Key: KAFKA-3904 URL: https://issues.apache.org/jira/browse/KAFKA-3904 Project: Kafka

[jira] [Work started] (KAFKA-3890) Kafka Streams: task assignment is not maintained on cluster restart or rolling restart

[ https://issues.apache.org/jira/browse/KAFKA-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3890 started by Henry Cai. > Kafka Streams: task assignment is not maintained on cluster restart or > r

[jira] [Commented] (KAFKA-3890) Kafka Streams: task assignment is not maintained on cluster restart or rolling restart

[ https://issues.apache.org/jira/browse/KAFKA-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15343209#comment-15343209 ] Henry Cai commented on KAFKA-3890: -- PR: https://github.com/apache/kafka/pull/

[jira] [Updated] (KAFKA-3890) Kafka Streams: task assignment is not maintained on cluster restart or rolling restart

[ https://issues.apache.org/jira/browse/KAFKA-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Cai updated KAFKA-3890: - Description: Currently the task assignment in TaskAssignor is not deterministic. During cluster restart

[jira] [Created] (KAFKA-3890) Kafka Streams: task assignment is not maintained on cluster restart or rolling restart

Henry Cai created KAFKA-3890: Summary: Kafka Streams: task assignment is not maintained on cluster restart or rolling restart Key: KAFKA-3890 URL: https://issues.apache.org/jira/browse/KAFKA-3890 Project

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[ https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15343115#comment-15343115 ] Henry Cai commented on KAFKA-3740: -- Yes, you or Roger can pick it up. > Add conf

Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

+1 On Thu, Jun 16, 2016 at 3:46 PM, Ismael Juma wrote: > +1 (binding) > > On Fri, Jun 17, 2016 at 12:44 AM, Guozhang Wang > wrote: > > > ＋1. > > > > On Thu, Jun 16, 2016 at 11:44 AM, Jason Gustafson > > wrote: > > > > > Hi All, > > > > > > I'd like to open the vote for KIP-62. This proposal at

Re: [DISCUSS] Java 8 as a minimum requirement

+1 for Lambda expression. On Thu, Jun 16, 2016 at 1:48 PM, Rajiv Kurian wrote: > +1 > > On Thu, Jun 16, 2016 at 1:45 PM, Ismael Juma wrote: > > > Hi all, > > > > I would like to start a discussion on making Java 8 a minimum requirement > > for Kafka's next feature release (let's say Kafka 0.10.

[jira] [Commented] (KAFKA-3059) ConsumerGroupCommand should allow resetting offsets for consumer groups

[ https://issues.apache.org/jira/browse/KAFKA-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332649#comment-15332649 ] Henry Cai commented on KAFKA-3059: -- We would like to have the capability to move

Re: [DISCUSS] KIP-63: Unify store and downstream caching in streams

One more thing for this KIP: Currently RocksDBWindowStore serialize the key/value before it puts into the in-memory cache, I think we should delay this serialization/deserialization unless it needs flush to db. For a simple countByKey for 100 records, this would trigger 100 serialization/deserial

Re: [DISCUSS] KIP-63: Unify store and downstream caching in streams

If we are going with a global setting, can the cache just be one gigantic LRU cache instead of many smaller ones? (partition_name, task_name) can be part of cache key. If there are many smaller caches, it's hard to achieve efficient global resource utilization (some caches are busy, some caches a

Re: [DISCUSS] KIP-62: Allow consumer to send heartbeats from a background thread

I have a question on the KIP on long stall during ProcessorStateManager.restoreActiveState(), this can be a long stall when we need to rebuild the RocksDB state on a new node. 1. Is restoreActiveState() considered as post rebalance since this is invoked on application rebalance listener? 2. When t

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[ https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303428#comment-15303428 ] Henry Cai commented on KAFKA-3740: -- Looks like you want two sets of RocksDB sett

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[ https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294310#comment-15294310 ] Henry Cai commented on KAFKA-3740: -- Worked now. > Add configs for RocksD

[jira] [Assigned] (KAFKA-3740) Add configs for RocksDBStores

[ https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Cai reassigned KAFKA-3740: Assignee: Henry Cai > Add configs for RocksDBSto

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[ https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294270#comment-15294270 ] Henry Cai commented on KAFKA-3740: -- Don't see that 'assign' button

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[ https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294204#comment-15294204 ] Henry Cai commented on KAFKA-3740: -- You can assign this one to me. > Add conf

[jira] [Commented] (KAFKA-3185) Allow users to cleanup internal data

[ https://issues.apache.org/jira/browse/KAFKA-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15267619#comment-15267619 ] Henry Cai commented on KAFKA-3185: -- Second to this. It's very painful for u

[jira] [Commented] (KAFKA-3101) Optimize Aggregation Outputs

[ https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15250977#comment-15250977 ] Henry Cai commented on KAFKA-3101: -- We really need this batching/buffering feature

[jira] [Updated] (KAFKA-3595) Add capability to specify replication compact option for stream store

[ https://issues.apache.org/jira/browse/KAFKA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Cai updated KAFKA-3595: - Description: Currently state store replication always go through a compact kafka topic. For some state

[jira] [Updated] (KAFKA-3596) Kafka Streams: Window expiration needs to consider more than event time

[ https://issues.apache.org/jira/browse/KAFKA-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Cai updated KAFKA-3596: - Description: Currently in Kafka Streams, the way the windows are expired in RocksDB is triggered by new

[jira] [Updated] (KAFKA-3595) Add capability to specify replication compact option for stream store

[ https://issues.apache.org/jira/browse/KAFKA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Cai updated KAFKA-3595: - Description: Currently in Kafka Streams, the way the windows are expired in RocksDB is triggered by new

[jira] [Created] (KAFKA-3596) Kafka Streams: Window expiration needs to consider more than event time

Henry Cai created KAFKA-3596: Summary: Kafka Streams: Window expiration needs to consider more than event time Key: KAFKA-3596 URL: https://issues.apache.org/jira/browse/KAFKA-3596 Project: Kafka

[jira] [Updated] (KAFKA-3595) Add capability to specify replication compact option for stream store

[ https://issues.apache.org/jira/browse/KAFKA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Cai updated KAFKA-3595: - Description: Currently state store replication always go through a compact kafka topic. For some state

[jira] [Updated] (KAFKA-3595) Add capability to specify replication compact option for stream store

[ https://issues.apache.org/jira/browse/KAFKA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Cai updated KAFKA-3595: - Affects Version/s: (was: 0.9.0.1) 0.10.1.0 > Add capability to spec

[jira] [Created] (KAFKA-3595) Add capability to specify replication compact option for stream store

Henry Cai created KAFKA-3595: Summary: Add capability to specify replication compact option for stream store Key: KAFKA-3595 URL: https://issues.apache.org/jira/browse/KAFKA-3595 Project: Kafka

[jira] [Updated] (KAFKA-3595) Add capability to specify replication compact option for stream store

[ https://issues.apache.org/jira/browse/KAFKA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Cai updated KAFKA-3595: - Issue Type: Improvement (was: New Feature) > Add capability to specify replication compact option