Re: Add a customized logo for Kafka Streams

I also like this one! Jiangjie (Becket) Qin On Fri, Mar 13, 2020 at 9:07 AM Matthias J. Sax wrote: > I personally love it! > > -Matthias > > On 3/12/20 11:31 AM, Sophie Blee-Goldman wrote: > > How reasonable of it. Let's try this again: > > Streams Lo

Re: [ANNOUNCE] New Kafka PMC member: Sriharsh Chintalapan

Congrats, Harsh! On Tue, Apr 23, 2019 at 5:41 AM Colin McCabe wrote: > Congratulations, Harsh! > > cheers, > Colin > > > On Thu, Apr 18, 2019, at 11:46, Jun Rao wrote: > > Hi, Everyone, > > > > Sriharsh Chintalapan has been active in the Kafka community since he > became > > a Kafka committer

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

Congrats, Matthias! On Sat, Apr 20, 2019 at 10:28 AM Matthias J. Sax wrote: > Thank you all! > > -Matthias > > > On 4/19/19 3:58 PM, Lei Chen wrote: > > Congratulations Matthias! Well deserved! > > > > -Lei > > > > On Fri, Apr 19, 2019 at 2:55 PM James Cheng > >

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

: compression.configs="gzip.compression.level:5, lz4.compression.level:17, zstd.compression.level:22" Thanks, Jiangjie (Becket) Qin On Thu, Feb 7, 2019 at 11:28 PM Dongjin Lee wrote: > Hi Becket, > > Understood. You are right, introducing map type is just a clearer way, but > not a stron

Re: [ANNOUNCE] New Committer: Randall Hauch

Congratulations, Randall! On Sat, Feb 16, 2019 at 2:44 AM Matthias J. Sax wrote: > Congrats Randall! > > > -Matthias > > On 2/14/19 6:16 PM, Guozhang Wang wrote: > > Hello all, > > > > The PMC of Apache Kafka is happy to announce another new committer > joining > > the project today: we have

Re: [ANNOUNCE] New Committer: Bill Bejeck

Congratulations, Bill! On Sat, Feb 16, 2019 at 1:57 AM Dong Lin wrote: > Congratulations Bill!! > > On Wed, Feb 13, 2019 at 5:03 PM Guozhang Wang wrote: > > > Hello all, > > > > The PMC of Apache Kafka is happy to announce that we've added Bill Bejeck > > as our newest project committer. > > >

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

igurations, whether to have a Map config type or not seems not that important. Given that the existing configuration uses the format of "KEY:VALUE[,KEY:VALUE]", we could just use this string format. Thanks, Jiangjie (Becket) Qin On Thu, Jan 31, 2019 at 7:47 PM Dongjin Lee wrote: >

Re: [VOTE] KIP-390: Allow fine-grained configuration for compression

Hi Dongjin, Thanks for the KIP. Sorry for being a little late on this, but I have some comments and have left them in the discussion thread to avoid polluting the voting thread. Could you take a look? Thanks, Jiangjie (Becket) Qin On Mon, Jan 21, 2019 at 3:17 PM Dongjin Lee wrote: > Hi

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

ession type. 2. Even for the same config, different compression type may have different terminologies. With the approach we can honor those terminologies instead of shoehorning them into the same configuration name. What do you think? Thanks, Jiangjie (Becket) Qin On Wed, Jan 23, 2019 at 12:07 A

Re: [VOTE] KIP-382 MirrorMaker 2.0

Hi Ryanne, Thanks for the KIP and patient discussion. +1 from me as well. Jiangjie (Becket) Qin On Fri, Jan 11, 2019 at 1:11 AM Jun Rao wrote: > Hi, Ryanne, > > Thanks for the explanation. All make sense to me now. +1 on the KIP from > me. > > Jun > > On Wed, Jan 9

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

the improvements of cross-cluster failover to a later decision. Thanks, Jiangjie (Becket) Qin On Tue, Jan 8, 2019 at 2:23 AM Ryanne Dolan wrote: > Hi Ewen, thanks for the questions. > > > On the ACL management, can you explain how things are supposed to work... > > There are t

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

will mirror the timestamps from source to target without being changed) Thanks, Jiangjie (Becket) Qin On Thu, Dec 27, 2018 at 1:16 AM Ryanne Dolan wrote: > Becket, this is great feedback, thanks. > > > having a reserved character for the topics is probably something worth > d

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

(target offsets = 199 + (150 -99) = 250) If the target offset is greater than the log end offset of the partition, that means that message has not been mirrored yet. Thanks, Jiangjie (Becket) Qin On Thu, Dec 20, 2018 at 6:00 AM Ryanne Dolan wrote: > Becket, thanks for taking a look. > &

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

on offset? If so how would that be done? Thanks, Jiangjie (Becket) Qin On Thu, Dec 13, 2018 at 8:23 AM Ryanne Dolan wrote: > > Consuming system restarts and restates from compacted topics, using > *.account_state > > Michael, I think I understand your concern. With MM2's remote t

Re: [DISCUSS] KIP-380: Detect outdated control requests and bounced brokers using broker generation

, Jiangjie (Becket) Qin On Tue, Nov 13, 2018 at 2:04 AM Dong Lin wrote: > Hey Becket, Patrick, > > Currently we expect that controller can only receive ControlledShutdownRequest > with outdated broker epoch in two cases: 1) the ControlledShutdownRequest > was sent by the broke

Re: [DISCUSS] KIP-380: Detect outdated control requests and bounced brokers using broker generation

in that case? Thanks, Jiangjie (Becket) Qin On Fri, Nov 9, 2018 at 2:17 AM Patrick Huang wrote: > Hi, > > In this KIP, we are also going to add a new exception and a new error code > "STALE_BROKER_EPOCH" in order to allow the broker to respond back the right > error when it s

Re: [VOTE] KIP-374: Add '--help' option to all available Kafka CLI commands

Thanks for the KIP. +1 (binding). On Mon, Nov 12, 2018 at 9:59 AM Harsha Chintalapani wrote: > +1 (binding) > > -Harsha > On Nov 11, 2018, 3:49 PM -0800, Daniele Ascione , > wrote: > > +1 (non-binding) > > > > Il ven 9 nov 2018, 02:09 Colin McCabe ha scritto: > > > > > +1 (binding) > > > > > >

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

the requests sent by the controller when the cluster is under heavy load. This leads to the issues Lucas mentioned in the motivation part. Thanks, Jiangjie (Becket) Qin > On Aug 20, 2018, at 11:33 PM, Eno Thereska wrote: > > Hi folks, > > I looked at the previous numbers that

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

Congrats, Dong! > On Aug 21, 2018, at 11:03 PM, Eno Thereska wrote: > > Congrats Dong! > > Eno > > On Tue, Aug 21, 2018 at 7:05 AM, Ted Yu wrote: > >> Congratulation Dong! >> >> On Tue, Aug 21, 2018 at 1:59 AM Viktor Somogyi-Vass < >> viktorsomo...@gmail.com> >> wrote: >> >>> Congrats

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

ead of doing it in different ways. Thanks, Jiangjie (Becket) Qin On Fri, Aug 17, 2018 at 4:34 AM, Lucas Wang wrote: > Thanks for the review, Becket. > > (1) After comparing the two approaches, I still feel the current writeup is > a little better. > a. The current writeup as

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

ker-lister-name to be "INTERNAL", then it knows to use the host name "host1.example.com", port 9093 and the security protocol PLAINTEXT to connect to the broker. should the port be 9092 instead? Thanks, Jiangjie (Becket) Qin On Tue, Aug 14, 2018 at 5:06 AM, Lucas Wang wrote: &

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

alternatives. Thanks, Jiangjie (Becket) Qin On Fri, Aug 10, 2018 at 6:23 AM, Lucas Wang wrote: > @Becket, > > I've asked for review by Jun and Joel in the vote thread. > Regarding the separate thread and port, I did talk about it in the rejected > alternative design 1. >

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

talking about the separate queue and not fully cover the changes we make now, e.g. separate thread, port, etc. We might want to explain a bit more so for people who did not follow the discussion mail thread also understand the whole proposal. Thanks, Jiangjie (Becket) Qin On Wed, Aug 8, 2018 at 12

Re: [DISCUSS] KIP-333 Consider a faster form of rebalancing

(Becket) Qin On Wed, Jul 18, 2018 at 2:39 PM, Richard Yu wrote: > Hi Becket, > I made some changes and clarified the motivation for this KIP. :)It should > be easier to understand now since I included a diagram. > Thanks,Richard Yu > On Tuesday, July 17, 2018, 4:38:11 PM GM

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Thanks for the updated KIP wiki, Lucas. Looks good to me overall. It might be an implementation detail, but do we still plan to use the correlation id to ensure the request processing order? Thanks, Jiangjie (Becket) Qin On Tue, Jul 31, 2018 at 3:39 AM, Lucas Wang wrote: > Thanks for y

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hi Lucas, Yes, I agree that a dedicated end to end control flow would be ideal. Thanks, Jiangjie (Becket) Qin On Tue, Jul 24, 2018 at 1:05 PM, Lucas Wang wrote: > Thanks for the comment, Becket. > So far, we've been trying to avoid making any request handler thread > special. &g

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

controller request queue and a dedicated controller request handler thread. Thanks, Jiangjie (Becket) Qin On Tue, Jul 24, 2018 at 8:16 AM, Lucas Wang wrote: > Sure, I can summarize the usage of correlation id. But before I do that, it > seems > the same out-of-order processing can al

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

processes it in the reverse order, the replica may still be wrongly recreated, right? Thanks, Jiangjie (Becket) Qin > On Jul 22, 2018, at 11:47 AM, Jun Rao wrote: > > Hmm, since we already use controller epoch and leader epoch for properly > caching the latest partition state, do we

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

thread controller request handler, the logic should be: 1. Process what ever request seen in the controller request queue 2. For the given epoch, drop request if its correlation id is smaller than that of the last processed request. Thanks, Jiangjie (Becket) Qin On Fri, Jul 20, 2018 at 8:07 AM, J

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

; the request queue almost at the same time. > > - So R1_a and R2 are processed in parallel. There is chance that R2's > > processing is completed before R1. > > > > If out-of-order processing can happen for both approaches with very low > > probability, it may not be

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

will be processed before R1 is processed, which may cause problem. Thanks, Jiangjie (Becket) Qin On Thu, Jul 19, 2018 at 3:23 AM, Joel Koshy wrote: > @Mayuresh - I like your idea. It appears to be a simpler less invasive > alternative and it should work. Jun/Becket/others, do you see any pi

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hey Joel, Thank for the detail explanation. I agree the current design makes sense. My confusion is about whether the new config for the controller queue capacity is necessary. I cannot think of a case in which users would change it. Thanks, Jiangjie (Becket) Qin On Wed, Jul 18, 2018 at 6:00

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Jiangjie (Becket) Qin On Wed, Jul 18, 2018 at 2:29 AM, Lucas Wang wrote: > @Becket > 1. Thanks for the comment. You are right that normally there should be just > one controller request because of muting, > and I had NOT intended to say there would be many enqueued controller > r

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

something very wrong has happened. Thanks, Jiangjie (Becket) Qin On Fri, Jul 13, 2018 at 1:10 PM, Dong Lin wrote: > Thanks for the update Lucas. > > I think the motivation section is intuitive. It will be good to learn more > about the comments from other reviewers. > > On Thu, Jul 1

Re: [DISCUSS] KIP-333 Consider a faster form of rebalancing

. If the application just wants to read from the end of the log after recovery from crash, would calling seekToEnd explicitly work? Thanks, Jiangjie (Becket) Qin On Thu, Jul 5, 2018 at 6:46 PM, Richard Yu wrote: > Hi all, > > I would like to discuss KIP-333 (which proposes a fa

Re: [VOTE] KIP-219 - Improve Quota Communication

the behavior dependency between different requests, which seems unnecessary. Due to the above reasons, I think it might be better to bump up all the request versions. Thanks, Jiangjie (Becket) Qin On Wed, Apr 25, 2018 at 3:19 PM, Jonghyun Lee <jonghy...@gmail.com> wrote: > Hello, > &g

Re: [DISCUSS] KIP-286: producer.send() should not block on metadata update

a performance bottleneck? Thanks, Jiangjie (Becket) Qin On Thu, Apr 12, 2018 at 1:41 AM, Dong Lin <lindon...@gmail.com> wrote: > Hey Ted, > > Thanks for your comments. With the proposed solution in the KIP, the memory > is only allocated once for the given message, which is the same

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

ee whether this can be improved. It would be at least one step forward to remove the usage of case 3. Thanks, Jiangjie (Becket) Qin On Mon, Mar 26, 2018 at 5:50 PM, Guozhang Wang <wangg...@gmail.com> wrote: > @Richard: TimeoutException inherits from RetriableException which inherits >

[ANNOUNCE] New Committer: Dong Lin

discussions and doing code reviews. Congratulations and looking forward to your future contribution, Dong! Jiangjie (Becket) Qin, on behalf of Apache Kafka PMC

Re: Streams meetup@LinkedIn (Mar 21)

Software Engineer, LinkedIn) 7:55-8:30 PM: Building Venice with Apache Kafka & Samza (Gaojie Liu, Senior Software Engineer, LinkedIn) Please use the following link to RSVP if you are interested. https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/248309045/ Thanks, Jiangjie (Becket)

Streams meetup@LinkedIn (Mar 21)

the details below if you are interested. https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/248309045/ <https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/248309045/> Thanks, Jiangjie (Becket) Qin

Re: 1.1 release progress

Hi Ismael, Yes, I am working on the fix. Will submit patch today. Thanks, Jiangjie (Becket) Qin On Thu, Feb 15, 2018 at 2:53 PM, Ismael Juma <ism...@juma.me.uk> wrote: > Hi Becket, > > Thanks for filing that. Are you working on a fix? > > Ismael > > On Thu, Feb 15,

Re: 1.1 release progress

Hi Damian, I just created another ticket KAFKA-6568, which I believe should also be a blocker unless people disagree. Thanks, Jiangjie (Becket) Qin On Wed, Feb 14, 2018 at 8:52 AM, Damian Guy <damian@gmail.com> wrote: > Hi All, > > The first 1.1 RC is due to be cut, howev

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

is essentially a performance improvement on top of this KIP. That can probably be done separately. Thanks, Jiangjie (Becket) Qin On Mon, Jan 29, 2018 at 11:52 AM, Dong Lin <lindon...@gmail.com> wrote: > Hey Colin, > > On Mon, Jan 29, 2018 at 11:23 AM, Colin McCabe <cmcc...

Re: [VOTE] KIP-219 - Improve Quota Communication

(binding) > > Regards, > > Rajini > > On Wed, Jan 17, 2018 at 5:24 PM, Becket Qin <becket@gmail.com> wrote: > > > Actually returning an empty fetch request may still be useful to reduce > the > > throttle time due to request quota violation because the Fet

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congratulations, Rajini! On Wed, Jan 17, 2018 at 1:52 PM, Ismael Juma wrote: > Congratulations Rajini! > > On 17 Jan 2018 10:49 am, "Gwen Shapira" wrote: > > Dear Kafka Developers, Users and Fans, > > Rajini Sivaram became a committer in April 2017. Since

Re: [VOTE] KIP-219 - Improve Quota Communication

Actually returning an empty fetch request may still be useful to reduce the throttle time due to request quota violation because the FetchResponse send time will be less. I just updated the KIP. Rajini, does that address your concern? On Tue, Jan 16, 2018 at 7:01 PM, Becket Qin <bec

Re: [VOTE] KIP-219 - Improve Quota Communication

Actually returning an empty fetch request may still be useful to reduce the throttle time due to request quota violation because the FetchResponse send time will be less. I just updated the KIP. Rajini, does that address your concern? On Tue, Jan 16, 2018 at 7:01 PM, Becket Qin <bec

Re: [VOTE] KIP-219 - Improve Quota Communication

FetchResponse or an empty one. Would it be more consistent if we throttle based on the actual throttle time / channel mute time? Thanks, Jiangjie (Becket) Qin On Tue, Jan 16, 2018 at 3:45 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Jiangjie, > > You are right that the he

Re: [ANNOUNCE] New committer: Matthias J. Sax

Congrats, Matthias! On Mon, Jan 15, 2018 at 9:54 PM, Konstantine Karantasis < konstant...@confluent.io> wrote: > Matthias! Congratulations! > > Konstantine > > On Mon, Jan 15, 2018 at 4:18 AM, Edoardo Comar wrote: > > > Congratulations Matthias ! > >

Re: [VOTE] KIP-219 - Improve Quota Communication

unless the request quota is hit, the rebalance won't be throttled. Even if request quota is hit, it seems unlikely to be delayed long. So it looks that we don't need to unmute the channel earlier than needed. Does this address your concern? Thanks, Jiangjie (Becket) Qin On Mon, Jan 15, 2018 at 4:22

Re: [VOTE] KIP-219 - Improve Quota Communication

Thanks for the comments, Jun. 1. Good point. 2. Also makes sense. Usually the connection.max.idle.ms is high enough so the throttling is impacted. I have updated the KIP to reflect the changes. Thanks, Jiangjie (Becket) Qin On Mon, Jan 8, 2018 at 6:30 PM, Jun Rao <j...@confluent.io>

Re: [VOTE] KIP-237: More Controller Health Metrics

+1. Thanks for the KIP. On Wed, Jan 3, 2018 at 9:28 AM, Manikumar wrote: > +1 (non-binding) > > Thanks > > On Wed, Jan 3, 2018 at 10:48 PM, Jun Rao wrote: > > > Hi, Dong, > > > > Thanks for the KIP. +1 > > > > Jun > > > > On Mon, Dec 18, 2017 at

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

want to replace them to avoid confusion. Thanks, Jiangjie (Becket) Qin On Wed, Jan 3, 2018 at 9:37 AM, Colin McCabe <cmcc...@apache.org> wrote: > On Tue, Jan 2, 2018, at 23:49, Becket Qin wrote: > > Thanks for the reply, Colin. > > > > My concern for the reinitia

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

for a complicated feature but implement them together. Perhaps we can do the same here if you want to limit the scope of this KIP. Thanks, Jiangjie (Becket) Qin On Tue, Jan 2, 2018 at 6:34 PM, Colin McCabe <cmcc...@apache.org> wrote: > On Tue, Jan 2, 2018, at 04:46, Becket Qin wrote: >

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi Colin, Good point about KIP-226. Maybe a separate broker epoch is needed although it is a little awkward to let the consumer set this. So was there a solution to the frequent pause and resume scenario? Did I miss something? Thanks, Jiangjie (Becket) Qin On Wed, Dec 27, 2017 at 1:40 PM

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

rmance numbers for prototypes. In any case, it's a separate problem which needs its >own KIP, I think. Those are indeed separate discussions. I was not intended to discuss them in this KIP. Sorry about the confusion. Thanks and Merry Christmas, Jiangjie (Becket) Qin On Sat, Dec 23, 2017 at 1:1

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

figuration knobs that aren't really necessary can harm usability. > > >>>> Certainly asking people to manually turn on a feature "where it > really > > >>>> hurts" seems to fall in that category, when we could easily enable > it > > >>&

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

to solve with broker epoch in KAFKA-6029. If we can make sure the session id will not change along the life time of clients, we can use that session id instead of creating a separate broker epoch and add that to the FetchRequest. Thanks, Jiangjie (Becket) Qin On Mon, Dec 11, 2017 at 3:25 PM

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

(Becket) Qin On Mon, Dec 11, 2017 at 1:06 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Colin, > > I went over the latest KIP wiki and have a few comments here. > > 1) The KIP says that client ID is a string if the session belongs to a > Kafka consumer. And it is

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

instance, a partition that's getting just one message a > second will appear in many fetch requests, unless every other partition > in the system is also only getting a low rate of incoming messages. > > regards, > Colin > > > > > Thanks, > > > > Jun &

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

the binary search logic already exists, it seems the additional object sanity check is not too much work. Not sure if the above implementation is simple enough to justify the performance improvement. Let me know if you see potential complexity. Thanks, Jiangjie (Becket) Qin On Wed, Dec 6, 2017

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

ng both offset and position because an index entry already contains offset and corresponding position. That is why we can save the binary search. Thanks, Jiangjie (Becket) Qin On Wed, Dec 6, 2017 at 11:20 AM, Colin McCabe <cmcc...@apache.org> wrote: > Hi Becket, > > Thanks for t

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

s supposed to be rare and if that happens usually it requires human attention to fix anyways. So reducing the regular cost in the normal cases probably makes more sense. Thanks, Jiangjie (Becket) Qin On Wed, Dec 6, 2017 at 10:58 AM, Colin McCabe <cmcc...@apache.org> wrote: > On Wed, Dec 6, 2017, a

Re: [VOTE] KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

+1, Thanks for the KIP. One thing is that we might want to make this in consistent with what proposed in KIP-225. On Wed, Dec 6, 2017 at 6:40 AM, Guozhang Wang wrote: > Hi Hu Xi, > > As I said before, it is only a clarification question for its internal > implementation; it

Re: [VOTE] KIP-225 - Use tags for consumer “records.lag” metrics

Hi Charly, Personally I prefer emitting both and deprecate old one. This does not block on the 2.0 release and we don't need to worry about more users picking up the old metric in 1.1 release. Thanks, Jiangjie (Becket) Qin On Tue, Dec 5, 2017 at 4:08 AM, charly molter <charly.mol...@gmail.

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

behavior impacting the latency due to page in/out of the index file. Thanks, Jiangjie (Becket) Qin On Tue, Dec 5, 2017 at 5:55 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Jiangjie, > > Not sure returning the fetch response at the index boundary is a general > solution.

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

there are continuous small requests going to each partition, which could be normal for some low latency systems. Thanks, Jiangjie (Becket) Qin On Tue, Dec 5, 2017 at 2:14 PM, Colin McCabe <cmcc...@apache.org> wrote: > On Tue, Dec 5, 2017, at 13:13, Jan Filipiak wrote: >

Re: [VOTE] KIP-225 - Use tags for consumer “records.lag” metrics

. Not sure if that is the plan or not. Thanks, Jiangjie (Becket) Qin On Mon, Dec 4, 2017 at 4:20 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Jiangjie, > > Since you proposed the original KIP-92, do you want to see if this KIP > makes sense? > > Thanks, > > Jun >

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

s, it might still be useful to let the session id be unique for each client instance (just like the producer id for the idempotent produce) and allow the client to update the leader side interested partitions and position with full FetchRequest without creating a new session id. Thanks, Jiangjie (

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

chRequest. Is there a scenario that only some of the partitions in a FetchResponse is lost? Thanks, Jiangjie (Becket) Qin On Sat, Dec 2, 2017 at 2:37 PM, Colin McCabe <cmcc...@apache.org> wrote: > On Fri, Dec 1, 2017, at 11:46, Dong Lin wrote: > > On Thu, Nov 30, 2017 at 9:37

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

(Becket) Qin On Thu, Nov 30, 2017 at 11:29 PM, Jan Filipiak <jan.filip...@trivago.com> wrote: > Hi, > > this discussion is going a little bit far from what I intended this thread > for. > I can see all of this beeing related. > > To let you guys know what I am currentl

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

of the traffic pattern in the cluster, In most cases the fetch request will be empty. 2. No need to maintain session epochs. What do you think? Thanks, Jiangjie (Becket) Qin On Thu, Nov 30, 2017 at 9:37 AM, Colin McCabe <cmcc...@apache.org> wrote: > On Wed, Nov 29, 2017, at 18:59, Dong

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

possible, it seems we can avoid the additional look up safely. Thanks, Jiangjie (Becket) Qin On Thu, Nov 23, 2017 at 9:31 AM, Jun Rao <j...@confluent.io> wrote: > Yes, caching the log segment position after the index lookup may work. One > subtle issue is that for a compacted topic, th

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

. Thanks, Jiangjie (Becket) Qin On Wed, Nov 22, 2017 at 4:53 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Colin, > > After step 3a, do we need to update the cached offset in the leader to be > the last offset in the data returned in the fetch response? If so, we need >

[VOTE] KIP-219 - Improve Quota Communication

+ communication The discussion thread is here: http://markmail.org/search/?q=kafka+KIP-219#query:kafka%20KIP-219+page:1+mid:ooxabguy7nz7l7zy+state:results Thanks, Jiangjie (Becket) Qin

Re: Stream Processing Meetup@LinkedIn (Dec 4th)

Hi Paolo, Yes, we will stream the meetup. Usually the link will be posted to the meetup website a couple of hours before the meetup. Feel free to ping me if you don't see it. Thanks, Jiangjie (Becket) Qin On Fri, Nov 17, 2017 at 11:59 AM, Paolo Patierno <ppatie...@live.com> wrote: >

Stream Processing Meetup@LinkedIn (Dec 4th)

Hi Kafka users and developers, We are going to host our quarterly Stream Processing Meetup@LinkedIn on Dec 4. There will be three speakers from Slack, Uber and LinkedIn. Please check the details below if you are interested. Thanks, Jiangjie (Becket) Qin *Stream Processing with Apache Kafka

Re: [DISCUSS] KIP-219 - Improve Quota Communication

, Jiangjie (Becket) Qin On Thu, Nov 16, 2017 at 11:47 AM, Rajini Sivaram <rajinisiva...@gmail.com> wrote: > Hi Becket, > > The current user quota doesn't solve the problem. But I was thinking that > if we could ensure we don't read more from the network than the quota > allows,

Re: [DISCUSS] KIP-219 - Improve Quota Communication

, the broker cannot sustain. In order to do that, we need to be able to throttle requests for more than request timeout, potentially a few minutes. It seems neither user quota nor limited throttle time can achieve this. Thanks, Jiangjie (Becket) Qin On Tue, Nov 14, 2017 at 7:44 AM, Rajini Sivaram

Re: [DISCUSS] KIP-219 - Improve Quota Communication

. But I am open to different opinions. Thanks, Jiangjie (Becket) Qin On Mon, Nov 6, 2017 at 7:28 PM, Becket Qin <becket@gmail.com> wrote: > Hi Jun, > > Hmm, even if a connection is closed by the client when the channel is > muted. After the channel is unmuted, it see

Re: [DISCUSS] KIP-219 - Improve Quota Communication

an arbitrarily long muted duration may indeed cause problem. Thanks, Jiangjie (Becket) Qin On Mon, Nov 6, 2017 at 7:22 PM, Becket Qin <becket@gmail.com> wrote: > Hi Rajini, > > Thanks for the detail explanation. Please see the reply below: > > 2. Limiting the throttle time to c

Re: [DISCUSS] KIP-219 - Improve Quota Communication

clients. Muting the channel is more of a migration plan so that we don't have regression on the old innocent (but good) clients. Thanks, Jiangjie (Becket) Qin On Mon, Nov 6, 2017 at 1:33 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Jiangjie, > > 3. If a client closes a socket con

Re: [ANNOUNCE] New committer: Onur Karaman

Congrats, Onur! On Mon, Nov 6, 2017 at 2:45 PM, James Cheng wrote: > Congrats Onur! Well deserved! > > -James > > > On Nov 6, 2017, at 9:24 AM, Jun Rao wrote: > > > > Hi, everyone, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka

Re: [DISCUSS] KIP-219 - Improve Quota Communication

the clients knows what happened so they can act accordingly instead of waiting naively. Muting the socket on the broker side is just in case of non-cooperating clients. For the existing clients, it seems this does not have much impact compare with what we have now. Thanks, Jiangjie (Becket) Qin

Re: [DISCUSS] KIP-219 - Improve Quota Communication

connection level quota, which is a bigger change and may be easier to discuss it in a separate KIP. Thanks, Jiangjie (Becket) Qin On Fri, Nov 3, 2017 at 10:28 AM, Jun Rao <j...@confluent.io> wrote: > Hi, Jiangjie, > > Thanks for bringing this up. A couple of quick thoughts. > > 1.

Re: [ANNOUNCE] Apache Kafka 1.0.0 Released

Great news! Thanks for running the release, Guozhang! On Thu, Nov 2, 2017 at 8:19 AM, Vahid S Hashemian wrote: > Great news. Thanks Guozhang! > > --Vahid > > > > > From: Rajini Sivaram > To: dev > Date:

[DISCUSS] KIP-219 - Improve Quota Communication

+quota+communication Comments are welcome. Thanks, Jiangjie (Becket) Qin

Re: [VOTE] KIP-214: Add zookeeper.max.in.flight.requests config to the broker

+1, Thanks for the KIP. On Mon, Oct 30, 2017 at 3:37 PM, Jeff Widman wrote: > +1 (non-binding) > > Thanks for putting the work in to benchmark various defaults. > > On Mon, Oct 30, 2017 at 3:05 PM, Ismael Juma wrote: > > > Thanks for the KIP, +1

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Paolo, Thanks for the KIP and sorry for being late on the thread. I am wondering what is the KafkaFuture returned by all() call? Should it be a Map<TopicPartition, Long> instead? Thanks, Jiangjie (Becket) QIn On Thu, Sep 28, 2017 at 3:48 AM, Paolo Patierno <ppatie...@live.com>

Re: [VOTE] KIP-192 : Provide cleaner semantics when idempotence is enabled

urgent to fix the upgrade path. Thanks, Jiangjie (Becket) Qin On Mon, Sep 11, 2017 at 4:13 PM, Apurva Mehta <apu...@confluent.io> wrote: > Hi Becket, > > Regarding the current implementation: we opted for a simpler server side > implementation where we _don't_ snapshot the me

Re: [VOTE] KIP-192 : Provide cleaner semantics when idempotence is enabled

f 5 max.in.flight.requests.per.connection 3. bounded memory footprint on the cached sequence/timestamp/offset entries. Hope it's not too late to have the changes if that makes sense. Thanks, Jiangjie (Becket) Qin On Mon, Sep 11, 2017 at 11:21 AM, Apurva Mehta <apu...@confluent.io> wrote:

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

+1. Thanks for the KIP, Sumant and Joel. On Fri, Sep 8, 2017 at 11:33 AM, Jason Gustafson wrote: > +1. Thanks for the KIP. > > On Fri, Sep 8, 2017 at 8:17 AM, Sumant Tambe wrote: > > > Updated. > > > > On 8 September 2017 at 02:04, Ismael Juma

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

(Becket) Qin On Wed, Sep 6, 2017 at 3:14 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Sumant, > > The diagram in the wiki seems to imply that delivery.timeout.ms doesn't > include the batching time. > > For retries, probably we can just default it to MAX_INT? > > Thank

Re: [DISCUSS] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

Sounds good to me as well. On Tue, Aug 29, 2017 at 2:43 AM, Ismael Juma wrote: > Sounds good to me too. Since this is a non controversial change, I suggest > starting the vote in 1-2 days if no-one else comments. > > Ismael > > On Thu, Aug 24, 2017 at 7:32 PM, Jason Gustafson

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

, but we will still wait for the response to be returned before sending the next request. Thanks, Jiangjie (Becket) Qin On Tue, Aug 29, 2017 at 4:00 PM, Jun Rao <j...@confluent.io> wrote: > Hmm, I thought delivery.timeout.ms bounds the time from a message is in > the > accumula

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

currently when TimeoutException is thrown, there is no guarantee whether the messages are delivered or not. Thanks, Jiangjie (Becket) Qin On Tue, Aug 29, 2017 at 12:33 PM, Jason Gustafson <ja...@confluent.io> wrote: > I think I'm with Becket. We should wait for request.timeout.ms

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

delivery.timeout.ms in this case is that if there are many batches to be expired in the queue, we may end up with continuous expirations and PID reset. Thanks, Jiangjie (Becket) Qin On Sun, Aug 27, 2017 at 12:08 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Jiangjie, > > If we

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

of min(remaining delivery.timeout.ms, request.timeout.ms)? Thanks, Jiangjie (Becket) Qin On Fri, Aug 25, 2017 at 9:34 AM, Jun Rao <j...@confluent.io> wrote: > Hi, Becket, > > Good point on expiring inflight requests. Perhaps we can expire an inflight > request a

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

the in-flight batch immediately, but wait for the produce response. If the batch has been successfully appended, we do not expire it. Otherwise, we expire it. Thanks, Jiangjie (Becket) Qin On Thu, Aug 24, 2017 at 11:26 AM, Jason Gustafson <ja...@confluent.io> wrote: > @Becket > > G

  1   2   3   4   >