Hi Dong, Sure. Regarding the 2nd case you mentioned "- If the controller has not received response for R1 before it is disconnected, it will re-send R1 followed by R2 after it is re-connected to the broker."
with the max inflight request set to 1, after the connection is re-established, the controller won't send R2 before it gets a response for R1, right? Plus the controller is using blocking calls for each request, i.e. NetworkClientUtils.sendAndReceive, with infinite retries for each request within the same instance of RequestSendThread. So within the same instance of RequestSendThread, sending out multiple different requests seems impossible. However, based on the comments in the source code, it seems multiple requests can happen if the broker loses its zk session, and then reconnects with zookeeper, multiple generations of RequestSendThreads can trigger multiple different requests. In that case, we cannot prevent out-of-order processing even with the queue since those multiple requests are from different connections. Broker generations can help in those cases, but I won't dive into that discussion. Is that right? Lucas On Wed, Jul 18, 2018 at 9:08 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Lucas, > > I think for now we can probably discuss based on the existing Kafka's > design where controller to a broker is hard coded to be 1. It looks like > Becket has provided a good example in which requests from the same > controller can be processed out of order. > > Thanks, > Dong > > On Wed, Jul 18, 2018 at 8:35 PM, Lucas Wang <lucasatu...@gmail.com> wrote: > > > @Becket and Dong, > > I think currently the ordering guarantee is achieved because > > the max inflight request from the controller to a broker is hard coded to > > be 1. > > > > If let's hypothetically say the max inflight requests is > 1, then I > think > > Dong > > is right to say that even the separate queue cannot guarantee ordered > > processing, > > For example, Req1 and Req2 are sent to a broker, and after a connection > > reconnection, > > both requests are sent again, causing the broker to have 4 requests in > the > > following order > > Req2 > Req1 > Req2 > Req1. > > > > In summary, it seems using the dequeue should not cause problems with > > out-of-order processing. > > Is that right? > > > > Lucas > > > > On Wed, Jul 18, 2018 at 6:24 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > > Hey Becket, > > > > > > It seems that the requests from the old controller will be discarded > due > > to > > > old controller epoch. It is not clear whether this is a problem. > > > > > > And if this out-of-order processing of controller requests is a > problem, > > it > > > seems like an existing problem which also applies to the multi-queue > > based > > > design. So it is probably not a concern specific to the use of deque. > > Does > > > that sound reasonable? > > > > > > Thanks, > > > Dong > > > > > > > > > On Wed, 18 Jul 2018 at 6:17 PM Becket Qin <becket....@gmail.com> > wrote: > > > > > > > Hi Mayuresh/Joel, > > > > > > > > Using the request channel as a dequeue was bright up some time ago > when > > > we > > > > initially thinking of prioritizing the request. The concern was that > > the > > > > controller requests are supposed to be processed in order. If we can > > > ensure > > > > that there is one controller request in the request channel, the > order > > is > > > > not a concern. But in cases that there are more than one controller > > > request > > > > inserted into the queue, the controller request order may change and > > > cause > > > > problem. For example, think about the following sequence: > > > > 1. Controller successfully sent a request R1 to broker > > > > 2. Broker receives R1 and put the request to the head of the request > > > queue. > > > > 3. Controller to broker connection failed and the controller > > reconnected > > > to > > > > the broker. > > > > 4. Controller sends a request R2 to the broker > > > > 5. Broker receives R2 and add it to the head of the request queue. > > > > Now on the broker side, R2 will be processed before R1 is processed, > > > which > > > > may cause problem. > > > > > > > > Thanks, > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > On Thu, Jul 19, 2018 at 3:23 AM, Joel Koshy <jjkosh...@gmail.com> > > wrote: > > > > > > > > > @Mayuresh - I like your idea. It appears to be a simpler less > > invasive > > > > > alternative and it should work. Jun/Becket/others, do you see any > > > > pitfalls > > > > > with this approach? > > > > > > > > > > On Wed, Jul 18, 2018 at 12:03 PM, Lucas Wang < > lucasatu...@gmail.com> > > > > > wrote: > > > > > > > > > > > @Mayuresh, > > > > > > That's a very interesting idea that I haven't thought before. > > > > > > It seems to solve our problem at hand pretty well, and also > > > > > > avoids the need to have a new size metric and capacity config > > > > > > for the controller request queue. In fact, if we were to adopt > > > > > > this design, there is no public interface change, and we > > > > > > probably don't need a KIP. > > > > > > Also implementation wise, it seems > > > > > > the java class LinkedBlockingQueue can readily satisfy the > > > requirement > > > > > > by supporting a capacity, and also allowing inserting at both > ends. > > > > > > > > > > > > My only concern is that this design is tied to the coincidence > that > > > > > > we have two request priorities and there are two ends to a deque. > > > > > > Hence by using the proposed design, it seems the network layer is > > > > > > more tightly coupled with upper layer logic, e.g. if we were to > add > > > > > > an extra priority level in the future for some reason, we would > > > > probably > > > > > > need to go back to the design of separate queues, one for each > > > priority > > > > > > level. > > > > > > > > > > > > In summary, I'm ok with both designs and lean toward your > suggested > > > > > > approach. > > > > > > Let's hear what others think. > > > > > > > > > > > > @Becket, > > > > > > In light of Mayuresh's suggested new design, I'm answering your > > > > question > > > > > > only in the context > > > > > > of the current KIP design: I think your suggestion makes sense, > and > > > I'm > > > > > ok > > > > > > with removing the capacity config and > > > > > > just relying on the default value of 20 being sufficient enough. > > > > > > > > > > > > Thanks, > > > > > > Lucas > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 18, 2018 at 9:57 AM, Mayuresh Gharat < > > > > > > gharatmayures...@gmail.com > > > > > > > wrote: > > > > > > > > > > > > > Hi Lucas, > > > > > > > > > > > > > > Seems like the main intent here is to prioritize the controller > > > > request > > > > > > > over any other requests. > > > > > > > In that case, we can change the request queue to a dequeue, > where > > > you > > > > > > > always insert the normal requests (produce, consume,..etc) to > the > > > end > > > > > of > > > > > > > the dequeue, but if its a controller request, you insert it to > > the > > > > head > > > > > > of > > > > > > > the queue. This ensures that the controller request will be > given > > > > > higher > > > > > > > priority over other requests. > > > > > > > > > > > > > > Also since we only read one request from the socket and mute it > > and > > > > > only > > > > > > > unmute it after handling the request, this would ensure that we > > > don't > > > > > > > handle controller requests out of order. > > > > > > > > > > > > > > With this approach we can avoid the second queue and the > > additional > > > > > > config > > > > > > > for the size of the queue. > > > > > > > > > > > > > > What do you think ? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Mayuresh > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 18, 2018 at 3:05 AM Becket Qin < > becket....@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > Hey Joel, > > > > > > > > > > > > > > > > Thank for the detail explanation. I agree the current design > > > makes > > > > > > sense. > > > > > > > > My confusion is about whether the new config for the > controller > > > > queue > > > > > > > > capacity is necessary. I cannot think of a case in which > users > > > > would > > > > > > > change > > > > > > > > it. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > On Wed, Jul 18, 2018 at 6:00 PM, Becket Qin < > > > becket....@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Lucas, > > > > > > > > > > > > > > > > > > I guess my question can be rephrased to "do we expect user > to > > > > ever > > > > > > > change > > > > > > > > > the controller request queue capacity"? If we agree that 20 > > is > > > > > > already > > > > > > > a > > > > > > > > > very generous default number and we do not expect user to > > > change > > > > > it, > > > > > > is > > > > > > > > it > > > > > > > > > still necessary to expose this as a config? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > On Wed, Jul 18, 2018 at 2:29 AM, Lucas Wang < > > > > lucasatu...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> @Becket > > > > > > > > >> 1. Thanks for the comment. You are right that normally > there > > > > > should > > > > > > be > > > > > > > > >> just > > > > > > > > >> one controller request because of muting, > > > > > > > > >> and I had NOT intended to say there would be many enqueued > > > > > > controller > > > > > > > > >> requests. > > > > > > > > >> I went through the KIP again, and I'm not sure which part > > > > conveys > > > > > > that > > > > > > > > >> info. > > > > > > > > >> I'd be happy to revise if you point it out the section. > > > > > > > > >> > > > > > > > > >> 2. Though it should not happen in normal conditions, the > > > current > > > > > > > design > > > > > > > > >> does not preclude multiple controllers running > > > > > > > > >> at the same time, hence if we don't have the controller > > queue > > > > > > capacity > > > > > > > > >> config and simply make its capacity to be 1, > > > > > > > > >> network threads handling requests from different > controllers > > > > will > > > > > be > > > > > > > > >> blocked during those troublesome times, > > > > > > > > >> which is probably not what we want. On the other hand, > > adding > > > > the > > > > > > > extra > > > > > > > > >> config with a default value, say 20, guards us from issues > > in > > > > > those > > > > > > > > >> troublesome times, and IMO there isn't much downside of > > adding > > > > the > > > > > > > extra > > > > > > > > >> config. > > > > > > > > >> > > > > > > > > >> @Mayuresh > > > > > > > > >> Good catch, this sentence is an obsolete statement based > on > > a > > > > > > previous > > > > > > > > >> design. I've revised the wording in the KIP. > > > > > > > > >> > > > > > > > > >> Thanks, > > > > > > > > >> Lucas > > > > > > > > >> > > > > > > > > >> On Tue, Jul 17, 2018 at 10:33 AM, Mayuresh Gharat < > > > > > > > > >> gharatmayures...@gmail.com> wrote: > > > > > > > > >> > > > > > > > > >> > Hi Lucas, > > > > > > > > >> > > > > > > > > > >> > Thanks for the KIP. > > > > > > > > >> > I am trying to understand why you think "The memory > > > > consumption > > > > > > can > > > > > > > > rise > > > > > > > > >> > given the total number of queued requests can go up to > 2x" > > > in > > > > > the > > > > > > > > impact > > > > > > > > >> > section. Normally the requests from controller to a > Broker > > > are > > > > > not > > > > > > > > high > > > > > > > > >> > volume, right ? > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > Thanks, > > > > > > > > >> > > > > > > > > > >> > Mayuresh > > > > > > > > >> > > > > > > > > > >> > On Tue, Jul 17, 2018 at 5:06 AM Becket Qin < > > > > > becket....@gmail.com> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > >> > > Thanks for the KIP, Lucas. Separating the control > plane > > > from > > > > > the > > > > > > > > data > > > > > > > > >> > plane > > > > > > > > >> > > makes a lot of sense. > > > > > > > > >> > > > > > > > > > > >> > > In the KIP you mentioned that the controller request > > queue > > > > may > > > > > > > have > > > > > > > > >> many > > > > > > > > >> > > requests in it. Will this be a common case? The > > controller > > > > > > > requests > > > > > > > > >> still > > > > > > > > >> > > goes through the SocketServer. The SocketServer will > > mute > > > > the > > > > > > > > channel > > > > > > > > >> > once > > > > > > > > >> > > a request is read and put into the request channel. So > > > > > assuming > > > > > > > > there > > > > > > > > >> is > > > > > > > > >> > > only one connection between controller and each > broker, > > on > > > > the > > > > > > > > broker > > > > > > > > >> > side, > > > > > > > > >> > > there should be only one controller request in the > > > > controller > > > > > > > > request > > > > > > > > >> > queue > > > > > > > > >> > > at any given time. If that is the case, do we need a > > > > separate > > > > > > > > >> controller > > > > > > > > >> > > request queue capacity config? The default value 20 > > means > > > > that > > > > > > we > > > > > > > > >> expect > > > > > > > > >> > > there are 20 controller switches to happen in a short > > > period > > > > > of > > > > > > > > time. > > > > > > > > >> I > > > > > > > > >> > am > > > > > > > > >> > > not sure whether someone should increase the > controller > > > > > request > > > > > > > > queue > > > > > > > > >> > > capacity to handle such case, as it seems indicating > > > > something > > > > > > > very > > > > > > > > >> wrong > > > > > > > > >> > > has happened. > > > > > > > > >> > > > > > > > > > > >> > > Thanks, > > > > > > > > >> > > > > > > > > > > >> > > Jiangjie (Becket) Qin > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > On Fri, Jul 13, 2018 at 1:10 PM, Dong Lin < > > > > > lindon...@gmail.com> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > > >> > > > Thanks for the update Lucas. > > > > > > > > >> > > > > > > > > > > > >> > > > I think the motivation section is intuitive. It will > > be > > > > good > > > > > > to > > > > > > > > >> learn > > > > > > > > >> > > more > > > > > > > > >> > > > about the comments from other reviewers. > > > > > > > > >> > > > > > > > > > > > >> > > > On Thu, Jul 12, 2018 at 9:48 PM, Lucas Wang < > > > > > > > > lucasatu...@gmail.com> > > > > > > > > >> > > wrote: > > > > > > > > >> > > > > > > > > > > > >> > > > > Hi Dong, > > > > > > > > >> > > > > > > > > > > > > >> > > > > I've updated the motivation section of the KIP by > > > > > explaining > > > > > > > the > > > > > > > > >> > cases > > > > > > > > >> > > > that > > > > > > > > >> > > > > would have user impacts. > > > > > > > > >> > > > > Please take a look at let me know your comments. > > > > > > > > >> > > > > > > > > > > > > >> > > > > Thanks, > > > > > > > > >> > > > > Lucas > > > > > > > > >> > > > > > > > > > > > > >> > > > > On Mon, Jul 9, 2018 at 5:53 PM, Lucas Wang < > > > > > > > > lucasatu...@gmail.com > > > > > > > > >> > > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > > > > > > > > > > >> > > > > > Hi Dong, > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > The simulation of disk being slow is merely for > me > > > to > > > > > > easily > > > > > > > > >> > > construct > > > > > > > > >> > > > a > > > > > > > > >> > > > > > testing scenario > > > > > > > > >> > > > > > with a backlog of produce requests. In > production, > > > > other > > > > > > > than > > > > > > > > >> the > > > > > > > > >> > > disk > > > > > > > > >> > > > > > being slow, a backlog of > > > > > > > > >> > > > > > produce requests may also be caused by high > > produce > > > > QPS. > > > > > > > > >> > > > > > In that case, we may not want to kill the broker > > and > > > > > > that's > > > > > > > > when > > > > > > > > >> > this > > > > > > > > >> > > > KIP > > > > > > > > >> > > > > > can be useful, both for JBOD > > > > > > > > >> > > > > > and non-JBOD setup. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Going back to your previous question about each > > > > > > > ProduceRequest > > > > > > > > >> > > covering > > > > > > > > >> > > > > 20 > > > > > > > > >> > > > > > partitions that are randomly > > > > > > > > >> > > > > > distributed, let's say a LeaderAndIsr request is > > > > > enqueued > > > > > > > that > > > > > > > > >> > tries > > > > > > > > >> > > to > > > > > > > > >> > > > > > switch the current broker, say broker0, from > > leader > > > to > > > > > > > > follower > > > > > > > > >> > > > > > *for one of the partitions*, say *test-0*. For > the > > > > sake > > > > > of > > > > > > > > >> > argument, > > > > > > > > >> > > > > > let's also assume the other brokers, say > broker1, > > > have > > > > > > > > *stopped* > > > > > > > > >> > > > fetching > > > > > > > > >> > > > > > from > > > > > > > > >> > > > > > the current broker, i.e. broker0. > > > > > > > > >> > > > > > 1. If the enqueued produce requests have acks = > > -1 > > > > > (ALL) > > > > > > > > >> > > > > > 1.1 without this KIP, the ProduceRequests > ahead > > of > > > > > > > > >> LeaderAndISR > > > > > > > > >> > > will > > > > > > > > >> > > > be > > > > > > > > >> > > > > > put into the purgatory, > > > > > > > > >> > > > > > and since they'll never be replicated to > > > other > > > > > > > brokers > > > > > > > > >> > > (because > > > > > > > > >> > > > > of > > > > > > > > >> > > > > > the assumption made above), they will > > > > > > > > >> > > > > > be completed either when the > LeaderAndISR > > > > > request > > > > > > is > > > > > > > > >> > > processed > > > > > > > > >> > > > or > > > > > > > > >> > > > > > when the timeout happens. > > > > > > > > >> > > > > > 1.2 With this KIP, broker0 will immediately > > > > transition > > > > > > the > > > > > > > > >> > > partition > > > > > > > > >> > > > > > test-0 to become a follower, > > > > > > > > >> > > > > > after the current broker sees the > > > replication > > > > of > > > > > > the > > > > > > > > >> > > remaining > > > > > > > > >> > > > 19 > > > > > > > > >> > > > > > partitions, it can send a response indicating > that > > > > > > > > >> > > > > > it's no longer the leader for the > > "test-0". > > > > > > > > >> > > > > > To see the latency difference between 1.1 and > > 1.2, > > > > > let's > > > > > > > say > > > > > > > > >> > there > > > > > > > > >> > > > are > > > > > > > > >> > > > > > 24K produce requests ahead of the LeaderAndISR, > > and > > > > > there > > > > > > > are > > > > > > > > 8 > > > > > > > > >> io > > > > > > > > >> > > > > threads, > > > > > > > > >> > > > > > so each io thread will process approximately > > 3000 > > > > > > produce > > > > > > > > >> > requests. > > > > > > > > >> > > > Now > > > > > > > > >> > > > > > let's investigate the io thread that finally > > > processed > > > > > the > > > > > > > > >> > > > LeaderAndISR. > > > > > > > > >> > > > > > For the 3000 produce requests, if we model the > > > time > > > > > when > > > > > > > > their > > > > > > > > >> > > > > remaining > > > > > > > > >> > > > > > 19 partitions catch up as t0, t1, ...t2999, and > > the > > > > > > > > LeaderAndISR > > > > > > > > >> > > > request > > > > > > > > >> > > > > is > > > > > > > > >> > > > > > processed at time t3000. > > > > > > > > >> > > > > > Without this KIP, the 1st produce request > would > > > have > > > > > > > waited > > > > > > > > an > > > > > > > > >> > > extra > > > > > > > > >> > > > > > t3000 - t0 time in the purgatory, the 2nd an > extra > > > > time > > > > > of > > > > > > > > >> t3000 - > > > > > > > > >> > > t1, > > > > > > > > >> > > > > etc. > > > > > > > > >> > > > > > Roughly speaking, the latency difference is > > bigger > > > > for > > > > > > the > > > > > > > > >> > earlier > > > > > > > > >> > > > > > produce requests than for the later ones. For > the > > > same > > > > > > > reason, > > > > > > > > >> the > > > > > > > > >> > > more > > > > > > > > >> > > > > > ProduceRequests queued > > > > > > > > >> > > > > > before the LeaderAndISR, the bigger benefit we > > get > > > > > > (capped > > > > > > > > by > > > > > > > > >> the > > > > > > > > >> > > > > > produce timeout). > > > > > > > > >> > > > > > 2. If the enqueued produce requests have acks=0 > or > > > > > acks=1 > > > > > > > > >> > > > > > There will be no latency differences in this > > case, > > > > but > > > > > > > > >> > > > > > 2.1 without this KIP, the records of partition > > > > test-0 > > > > > in > > > > > > > the > > > > > > > > >> > > > > > ProduceRequests ahead of the LeaderAndISR will > be > > > > > appended > > > > > > > to > > > > > > > > >> the > > > > > > > > >> > > local > > > > > > > > >> > > > > log, > > > > > > > > >> > > > > > and eventually be truncated after > > processing > > > > the > > > > > > > > >> > > LeaderAndISR. > > > > > > > > >> > > > > > This is what's referred to as > > > > > > > > >> > > > > > "some unofficial definition of data loss > > in > > > > > terms > > > > > > of > > > > > > > > >> > messages > > > > > > > > >> > > > > > beyond the high watermark". > > > > > > > > >> > > > > > 2.2 with this KIP, we can mitigate the effect > > > since > > > > if > > > > > > the > > > > > > > > >> > > > LeaderAndISR > > > > > > > > >> > > > > > is immediately processed, the response to > > producers > > > > will > > > > > > > have > > > > > > > > >> > > > > > the NotLeaderForPartition error, causing > > > > > producers > > > > > > > to > > > > > > > > >> retry > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > This explanation above is the benefit for > reducing > > > the > > > > > > > latency > > > > > > > > >> of a > > > > > > > > >> > > > > broker > > > > > > > > >> > > > > > becoming the follower, > > > > > > > > >> > > > > > closely related is reducing the latency of a > > broker > > > > > > becoming > > > > > > > > the > > > > > > > > >> > > > leader. > > > > > > > > >> > > > > > In this case, the benefit is even more obvious, > if > > > > other > > > > > > > > brokers > > > > > > > > >> > have > > > > > > > > >> > > > > > resigned leadership, and the > > > > > > > > >> > > > > > current broker should take leadership. Any delay > > in > > > > > > > processing > > > > > > > > >> the > > > > > > > > >> > > > > > LeaderAndISR will be perceived > > > > > > > > >> > > > > > by clients as unavailability. In extreme cases, > > this > > > > can > > > > > > > cause > > > > > > > > >> > failed > > > > > > > > >> > > > > > produce requests if the retries are > > > > > > > > >> > > > > > exhausted. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Another two types of controller requests are > > > > > > UpdateMetadata > > > > > > > > and > > > > > > > > >> > > > > > StopReplica, which I'll briefly discuss as > > follows: > > > > > > > > >> > > > > > For UpdateMetadata requests, delayed processing > > > means > > > > > > > clients > > > > > > > > >> > > receiving > > > > > > > > >> > > > > > stale metadata, e.g. with the wrong leadership > > info > > > > > > > > >> > > > > > for certain partitions, and the effect is more > > > retries > > > > > or > > > > > > > even > > > > > > > > >> > fatal > > > > > > > > >> > > > > > failure if the retries are exhausted. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > For StopReplica requests, a long queuing time > may > > > > > degrade > > > > > > > the > > > > > > > > >> > > > performance > > > > > > > > >> > > > > > of topic deletion. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Regarding your last question of the delay for > > > > > > > > >> > DescribeLogDirsRequest, > > > > > > > > >> > > > you > > > > > > > > >> > > > > > are right > > > > > > > > >> > > > > > that this KIP cannot help with the latency in > > > getting > > > > > the > > > > > > > log > > > > > > > > >> dirs > > > > > > > > >> > > > info, > > > > > > > > >> > > > > > and it's only relevant > > > > > > > > >> > > > > > when controller requests are involved. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Regards, > > > > > > > > >> > > > > > Lucas > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > On Tue, Jul 3, 2018 at 5:11 PM, Dong Lin < > > > > > > > lindon...@gmail.com > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > >> > > > > > > > > > > > > > >> > > > > >> Hey Jun, > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> Thanks much for the comments. It is good point. > > So > > > > the > > > > > > > > feature > > > > > > > > >> may > > > > > > > > >> > > be > > > > > > > > >> > > > > >> useful for JBOD use-case. I have one question > > > below. > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> Hey Lucas, > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> Do you think this feature is also useful for > > > non-JBOD > > > > > > setup > > > > > > > > or > > > > > > > > >> it > > > > > > > > >> > is > > > > > > > > >> > > > > only > > > > > > > > >> > > > > >> useful for the JBOD setup? It may be useful to > > > > > understand > > > > > > > > this. > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> When the broker is setup using JBOD, in order > to > > > move > > > > > > > leaders > > > > > > > > >> on > > > > > > > > >> > the > > > > > > > > >> > > > > >> failed > > > > > > > > >> > > > > >> disk to other disks, the system operator first > > > needs > > > > to > > > > > > get > > > > > > > > the > > > > > > > > >> > list > > > > > > > > >> > > > of > > > > > > > > >> > > > > >> partitions on the failed disk. This is > currently > > > > > achieved > > > > > > > > using > > > > > > > > >> > > > > >> AdminClient.describeLogDirs(), which sends > > > > > > > > >> DescribeLogDirsRequest > > > > > > > > >> > to > > > > > > > > >> > > > the > > > > > > > > >> > > > > >> broker. If we only prioritize the controller > > > > requests, > > > > > > then > > > > > > > > the > > > > > > > > >> > > > > >> DescribeLogDirsRequest > > > > > > > > >> > > > > >> may still take a long time to be processed by > the > > > > > broker. > > > > > > > So > > > > > > > > >> the > > > > > > > > >> > > > overall > > > > > > > > >> > > > > >> time to move leaders away from the failed disk > > may > > > > > still > > > > > > be > > > > > > > > >> long > > > > > > > > >> > > even > > > > > > > > >> > > > > with > > > > > > > > >> > > > > >> this KIP. What do you think? > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> Thanks, > > > > > > > > >> > > > > >> Dong > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> On Tue, Jul 3, 2018 at 4:38 PM, Lucas Wang < > > > > > > > > >> lucasatu...@gmail.com > > > > > > > > >> > > > > > > > > > > >> > > > > wrote: > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > Thanks for the insightful comment, Jun. > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > @Dong, > > > > > > > > >> > > > > >> > Since both of the two comments in your > previous > > > > email > > > > > > are > > > > > > > > >> about > > > > > > > > >> > > the > > > > > > > > >> > > > > >> > benefits of this KIP and whether it's useful, > > > > > > > > >> > > > > >> > in light of Jun's last comment, do you agree > > that > > > > > this > > > > > > > KIP > > > > > > > > >> can > > > > > > > > >> > be > > > > > > > > >> > > > > >> > beneficial in the case mentioned by Jun? > > > > > > > > >> > > > > >> > Please let me know, thanks! > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > Regards, > > > > > > > > >> > > > > >> > Lucas > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > On Tue, Jul 3, 2018 at 2:07 PM, Jun Rao < > > > > > > > j...@confluent.io> > > > > > > > > >> > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > Hi, Lucas, Dong, > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > If all disks on a broker are slow, one > > probably > > > > > > should > > > > > > > > just > > > > > > > > >> > kill > > > > > > > > >> > > > the > > > > > > > > >> > > > > >> > > broker. In that case, this KIP may not > help. > > If > > > > > only > > > > > > > one > > > > > > > > of > > > > > > > > >> > the > > > > > > > > >> > > > > disks > > > > > > > > >> > > > > >> on > > > > > > > > >> > > > > >> > a > > > > > > > > >> > > > > >> > > broker is slow, one may want to fail that > > disk > > > > and > > > > > > move > > > > > > > > the > > > > > > > > >> > > > leaders > > > > > > > > >> > > > > on > > > > > > > > >> > > > > >> > that > > > > > > > > >> > > > > >> > > disk to other brokers. In that case, being > > able > > > > to > > > > > > > > process > > > > > > > > >> the > > > > > > > > >> > > > > >> > LeaderAndIsr > > > > > > > > >> > > > > >> > > requests faster will potentially help the > > > > producers > > > > > > > > recover > > > > > > > > >> > > > quicker. > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > Thanks, > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > Jun > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > On Mon, Jul 2, 2018 at 7:56 PM, Dong Lin < > > > > > > > > >> lindon...@gmail.com > > > > > > > > >> > > > > > > > > > > >> > > > > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > Hey Lucas, > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > Thanks for the reply. Some follow up > > > questions > > > > > > below. > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > Regarding 1, if each ProduceRequest > covers > > 20 > > > > > > > > partitions > > > > > > > > >> > that > > > > > > > > >> > > > are > > > > > > > > >> > > > > >> > > randomly > > > > > > > > >> > > > > >> > > > distributed across all partitions, then > > each > > > > > > > > >> ProduceRequest > > > > > > > > >> > > will > > > > > > > > >> > > > > >> likely > > > > > > > > >> > > > > >> > > > cover some partitions for which the > broker > > is > > > > > still > > > > > > > > >> leader > > > > > > > > >> > > after > > > > > > > > >> > > > > it > > > > > > > > >> > > > > >> > > quickly > > > > > > > > >> > > > > >> > > > processes the > > > > > > > > >> > > > > >> > > > LeaderAndIsrRequest. Then broker will > still > > > be > > > > > slow > > > > > > > in > > > > > > > > >> > > > processing > > > > > > > > >> > > > > >> these > > > > > > > > >> > > > > >> > > > ProduceRequest and request will still be > > very > > > > > high > > > > > > > with > > > > > > > > >> this > > > > > > > > >> > > > KIP. > > > > > > > > >> > > > > It > > > > > > > > >> > > > > >> > > seems > > > > > > > > >> > > > > >> > > > that most ProduceRequest will still > timeout > > > > after > > > > > > 30 > > > > > > > > >> > seconds. > > > > > > > > >> > > Is > > > > > > > > >> > > > > >> this > > > > > > > > >> > > > > >> > > > understanding correct? > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > Regarding 2, if most ProduceRequest will > > > still > > > > > > > timeout > > > > > > > > >> after > > > > > > > > >> > > 30 > > > > > > > > >> > > > > >> > seconds, > > > > > > > > >> > > > > >> > > > then it is less clear how this KIP > reduces > > > > > average > > > > > > > > >> produce > > > > > > > > >> > > > > latency. > > > > > > > > >> > > > > >> Can > > > > > > > > >> > > > > >> > > you > > > > > > > > >> > > > > >> > > > clarify what metrics can be improved by > > this > > > > KIP? > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > Not sure why system operator directly > cares > > > > > number > > > > > > of > > > > > > > > >> > > truncated > > > > > > > > >> > > > > >> > messages. > > > > > > > > >> > > > > >> > > > Do you mean this KIP can improve average > > > > > throughput > > > > > > > or > > > > > > > > >> > reduce > > > > > > > > >> > > > > >> message > > > > > > > > >> > > > > >> > > > duplication? It will be good to > understand > > > > this. > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > Thanks, > > > > > > > > >> > > > > >> > > > Dong > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > On Tue, 3 Jul 2018 at 7:12 AM Lucas Wang > < > > > > > > > > >> > > lucasatu...@gmail.com > > > > > > > > >> > > > > > > > > > > > > >> > > > > >> > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > Hi Dong, > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > Thanks for your valuable comments. > Please > > > see > > > > > my > > > > > > > > reply > > > > > > > > >> > > below. > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > 1. The Google doc showed only 1 > > partition. > > > > Now > > > > > > > let's > > > > > > > > >> > > consider > > > > > > > > >> > > > a > > > > > > > > >> > > > > >> more > > > > > > > > >> > > > > >> > > > common > > > > > > > > >> > > > > >> > > > > scenario > > > > > > > > >> > > > > >> > > > > where broker0 is the leader of many > > > > partitions. > > > > > > And > > > > > > > > >> let's > > > > > > > > >> > > say > > > > > > > > >> > > > > for > > > > > > > > >> > > > > >> > some > > > > > > > > >> > > > > >> > > > > reason its IO becomes slow. > > > > > > > > >> > > > > >> > > > > The number of leader partitions on > > broker0 > > > is > > > > > so > > > > > > > > large, > > > > > > > > >> > say > > > > > > > > >> > > > 10K, > > > > > > > > >> > > > > >> that > > > > > > > > >> > > > > >> > > the > > > > > > > > >> > > > > >> > > > > cluster is skewed, > > > > > > > > >> > > > > >> > > > > and the operator would like to shift > the > > > > > > leadership > > > > > > > > >> for a > > > > > > > > >> > > lot > > > > > > > > >> > > > of > > > > > > > > >> > > > > >> > > > > partitions, say 9K, to other brokers, > > > > > > > > >> > > > > >> > > > > either manually or through some service > > > like > > > > > > cruise > > > > > > > > >> > control. > > > > > > > > >> > > > > >> > > > > With this KIP, not only will the > > leadership > > > > > > > > transitions > > > > > > > > >> > > finish > > > > > > > > >> > > > > >> more > > > > > > > > >> > > > > >> > > > > quickly, helping the cluster itself > > > becoming > > > > > more > > > > > > > > >> > balanced, > > > > > > > > >> > > > > >> > > > > but all existing producers > corresponding > > to > > > > the > > > > > > 9K > > > > > > > > >> > > partitions > > > > > > > > >> > > > > will > > > > > > > > >> > > > > >> > get > > > > > > > > >> > > > > >> > > > the > > > > > > > > >> > > > > >> > > > > errors relatively quickly > > > > > > > > >> > > > > >> > > > > rather than relying on their timeout, > > > thanks > > > > to > > > > > > the > > > > > > > > >> > batched > > > > > > > > >> > > > > async > > > > > > > > >> > > > > >> ZK > > > > > > > > >> > > > > >> > > > > operations. > > > > > > > > >> > > > > >> > > > > To me it's a useful feature to have > > during > > > > such > > > > > > > > >> > troublesome > > > > > > > > >> > > > > times. > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > 2. The experiments in the Google Doc > have > > > > shown > > > > > > > that > > > > > > > > >> with > > > > > > > > >> > > this > > > > > > > > >> > > > > KIP > > > > > > > > >> > > > > >> > many > > > > > > > > >> > > > > >> > > > > producers > > > > > > > > >> > > > > >> > > > > receive an explicit error > > > > > NotLeaderForPartition, > > > > > > > > based > > > > > > > > >> on > > > > > > > > >> > > > which > > > > > > > > >> > > > > >> they > > > > > > > > >> > > > > >> > > > retry > > > > > > > > >> > > > > >> > > > > immediately. > > > > > > > > >> > > > > >> > > > > Therefore the latency (~14 > seconds+quick > > > > retry) > > > > > > for > > > > > > > > >> their > > > > > > > > >> > > > single > > > > > > > > >> > > > > >> > > message > > > > > > > > >> > > > > >> > > > is > > > > > > > > >> > > > > >> > > > > much smaller > > > > > > > > >> > > > > >> > > > > compared with the case of timing out > > > without > > > > > the > > > > > > > KIP > > > > > > > > >> (30 > > > > > > > > >> > > > seconds > > > > > > > > >> > > > > >> for > > > > > > > > >> > > > > >> > > > timing > > > > > > > > >> > > > > >> > > > > out + quick retry). > > > > > > > > >> > > > > >> > > > > One might argue that reducing the > timing > > > out > > > > on > > > > > > the > > > > > > > > >> > producer > > > > > > > > >> > > > > side > > > > > > > > >> > > > > >> can > > > > > > > > >> > > > > >> > > > > achieve the same result, > > > > > > > > >> > > > > >> > > > > yet reducing the timeout has its own > > > > > > drawbacks[1]. > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > Also *IF* there were a metric to show > the > > > > > number > > > > > > of > > > > > > > > >> > > truncated > > > > > > > > >> > > > > >> > messages > > > > > > > > >> > > > > >> > > on > > > > > > > > >> > > > > >> > > > > brokers, > > > > > > > > >> > > > > >> > > > > with the experiments done in the Google > > > Doc, > > > > it > > > > > > > > should > > > > > > > > >> be > > > > > > > > >> > > easy > > > > > > > > >> > > > > to > > > > > > > > >> > > > > >> see > > > > > > > > >> > > > > >> > > > that > > > > > > > > >> > > > > >> > > > > a lot fewer messages need > > > > > > > > >> > > > > >> > > > > to be truncated on broker0 since the > > > > up-to-date > > > > > > > > >> metadata > > > > > > > > >> > > > avoids > > > > > > > > >> > > > > >> > > appending > > > > > > > > >> > > > > >> > > > > of messages > > > > > > > > >> > > > > >> > > > > in subsequent PRODUCE requests. If we > > talk > > > > to a > > > > > > > > system > > > > > > > > >> > > > operator > > > > > > > > >> > > > > >> and > > > > > > > > >> > > > > >> > ask > > > > > > > > >> > > > > >> > > > > whether > > > > > > > > >> > > > > >> > > > > they prefer fewer wasteful IOs, I bet > > most > > > > > likely > > > > > > > the > > > > > > > > >> > answer > > > > > > > > >> > > > is > > > > > > > > >> > > > > >> yes. > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > 3. To answer your question, I think it > > > might > > > > be > > > > > > > > >> helpful to > > > > > > > > >> > > > > >> construct > > > > > > > > >> > > > > >> > > some > > > > > > > > >> > > > > >> > > > > formulas. > > > > > > > > >> > > > > >> > > > > To simplify the modeling, I'm going > back > > to > > > > the > > > > > > > case > > > > > > > > >> where > > > > > > > > >> > > > there > > > > > > > > >> > > > > >> is > > > > > > > > >> > > > > >> > > only > > > > > > > > >> > > > > >> > > > > ONE partition involved. > > > > > > > > >> > > > > >> > > > > Following the experiments in the Google > > > Doc, > > > > > > let's > > > > > > > > say > > > > > > > > >> > > broker0 > > > > > > > > >> > > > > >> > becomes > > > > > > > > >> > > > > >> > > > the > > > > > > > > >> > > > > >> > > > > follower at time t0, > > > > > > > > >> > > > > >> > > > > and after t0 there were still N produce > > > > > requests > > > > > > in > > > > > > > > its > > > > > > > > >> > > > request > > > > > > > > >> > > > > >> > queue. > > > > > > > > >> > > > > >> > > > > With the up-to-date metadata brought by > > > this > > > > > KIP, > > > > > > > > >> broker0 > > > > > > > > >> > > can > > > > > > > > >> > > > > >> reply > > > > > > > > >> > > > > >> > > with > > > > > > > > >> > > > > >> > > > an > > > > > > > > >> > > > > >> > > > > NotLeaderForPartition exception, > > > > > > > > >> > > > > >> > > > > let's use M1 to denote the average > > > processing > > > > > > time > > > > > > > of > > > > > > > > >> > > replying > > > > > > > > >> > > > > >> with > > > > > > > > >> > > > > >> > > such > > > > > > > > >> > > > > >> > > > an > > > > > > > > >> > > > > >> > > > > error message. > > > > > > > > >> > > > > >> > > > > Without this KIP, the broker will need > to > > > > > append > > > > > > > > >> messages > > > > > > > > >> > to > > > > > > > > >> > > > > >> > segments, > > > > > > > > >> > > > > >> > > > > which may trigger a flush to disk, > > > > > > > > >> > > > > >> > > > > let's use M2 to denote the average > > > processing > > > > > > time > > > > > > > > for > > > > > > > > >> > such > > > > > > > > >> > > > > logic. > > > > > > > > >> > > > > >> > > > > Then the average extra latency incurred > > > > without > > > > > > > this > > > > > > > > >> KIP > > > > > > > > >> > is > > > > > > > > >> > > N > > > > > > > > >> > > > * > > > > > > > > >> > > > > >> (M2 - > > > > > > > > >> > > > > >> > > > M1) / > > > > > > > > >> > > > > >> > > > > 2. > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > In practice, M2 should always be larger > > > than > > > > > M1, > > > > > > > > which > > > > > > > > >> > means > > > > > > > > >> > > > as > > > > > > > > >> > > > > >> long > > > > > > > > >> > > > > >> > > as N > > > > > > > > >> > > > > >> > > > > is positive, > > > > > > > > >> > > > > >> > > > > we would see improvements on the > average > > > > > latency. > > > > > > > > >> > > > > >> > > > > There does not need to be significant > > > backlog > > > > > of > > > > > > > > >> requests > > > > > > > > >> > in > > > > > > > > >> > > > the > > > > > > > > >> > > > > >> > > request > > > > > > > > >> > > > > >> > > > > queue, > > > > > > > > >> > > > > >> > > > > or severe degradation of disk > performance > > > to > > > > > have > > > > > > > the > > > > > > > > >> > > > > improvement. > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > Regards, > > > > > > > > >> > > > > >> > > > > Lucas > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > [1] For instance, reducing the timeout > on > > > the > > > > > > > > producer > > > > > > > > >> > side > > > > > > > > >> > > > can > > > > > > > > >> > > > > >> > trigger > > > > > > > > >> > > > > >> > > > > unnecessary duplicate requests > > > > > > > > >> > > > > >> > > > > when the corresponding leader broker is > > > > > > overloaded, > > > > > > > > >> > > > exacerbating > > > > > > > > >> > > > > >> the > > > > > > > > >> > > > > >> > > > > situation. > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > On Sun, Jul 1, 2018 at 9:18 PM, Dong > Lin > > < > > > > > > > > >> > > lindon...@gmail.com > > > > > > > > >> > > > > > > > > > > > > >> > > > > >> > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > Hey Lucas, > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > Thanks much for the detailed > > > documentation > > > > of > > > > > > the > > > > > > > > >> > > > experiment. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > Initially I also think having a > > separate > > > > > queue > > > > > > > for > > > > > > > > >> > > > controller > > > > > > > > >> > > > > >> > > requests > > > > > > > > >> > > > > >> > > > is > > > > > > > > >> > > > > >> > > > > > useful because, as you mentioned in > the > > > > > summary > > > > > > > > >> section > > > > > > > > >> > of > > > > > > > > >> > > > the > > > > > > > > >> > > > > >> > Google > > > > > > > > >> > > > > >> > > > > doc, > > > > > > > > >> > > > > >> > > > > > controller requests are generally > more > > > > > > important > > > > > > > > than > > > > > > > > >> > data > > > > > > > > >> > > > > >> requests > > > > > > > > >> > > > > >> > > and > > > > > > > > >> > > > > >> > > > > we > > > > > > > > >> > > > > >> > > > > > probably want controller requests to > be > > > > > > processed > > > > > > > > >> > sooner. > > > > > > > > >> > > > But > > > > > > > > >> > > > > >> then > > > > > > > > >> > > > > >> > > Eno > > > > > > > > >> > > > > >> > > > > has > > > > > > > > >> > > > > >> > > > > > two very good questions which I am > not > > > sure > > > > > the > > > > > > > > >> Google > > > > > > > > >> > doc > > > > > > > > >> > > > has > > > > > > > > >> > > > > >> > > answered > > > > > > > > >> > > > > >> > > > > > explicitly. Could you help with the > > > > following > > > > > > > > >> questions? > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > 1) It is not very clear what is the > > > actual > > > > > > > benefit > > > > > > > > of > > > > > > > > >> > > > KIP-291 > > > > > > > > >> > > > > to > > > > > > > > >> > > > > >> > > users. > > > > > > > > >> > > > > >> > > > > The > > > > > > > > >> > > > > >> > > > > > experiment setup in the Google doc > > > > simulates > > > > > > the > > > > > > > > >> > scenario > > > > > > > > >> > > > that > > > > > > > > >> > > > > >> > broker > > > > > > > > >> > > > > >> > > > is > > > > > > > > >> > > > > >> > > > > > very slow handling ProduceRequest due > > to > > > > e.g. > > > > > > > slow > > > > > > > > >> disk. > > > > > > > > >> > > It > > > > > > > > >> > > > > >> > currently > > > > > > > > >> > > > > >> > > > > > assumes that there is only 1 > partition. > > > But > > > > > in > > > > > > > the > > > > > > > > >> > common > > > > > > > > >> > > > > >> scenario, > > > > > > > > >> > > > > >> > > it > > > > > > > > >> > > > > >> > > > is > > > > > > > > >> > > > > >> > > > > > probably reasonable to assume that > > there > > > > are > > > > > > many > > > > > > > > >> other > > > > > > > > >> > > > > >> partitions > > > > > > > > >> > > > > >> > > that > > > > > > > > >> > > > > >> > > > > are > > > > > > > > >> > > > > >> > > > > > also actively produced to and > > > > ProduceRequest > > > > > to > > > > > > > > these > > > > > > > > >> > > > > partition > > > > > > > > >> > > > > >> > also > > > > > > > > >> > > > > >> > > > > takes > > > > > > > > >> > > > > >> > > > > > e.g. 2 seconds to be processed. So > even > > > if > > > > > > > broker0 > > > > > > > > >> can > > > > > > > > >> > > > become > > > > > > > > >> > > > > >> > > follower > > > > > > > > >> > > > > >> > > > > for > > > > > > > > >> > > > > >> > > > > > the partition 0 soon, it probably > still > > > > needs > > > > > > to > > > > > > > > >> process > > > > > > > > >> > > the > > > > > > > > >> > > > > >> > > > > ProduceRequest > > > > > > > > >> > > > > >> > > > > > slowly t in the queue because these > > > > > > > ProduceRequests > > > > > > > > >> > cover > > > > > > > > >> > > > > other > > > > > > > > >> > > > > >> > > > > partitions. > > > > > > > > >> > > > > >> > > > > > Thus most ProduceRequest will still > > > timeout > > > > > > after > > > > > > > > 30 > > > > > > > > >> > > seconds > > > > > > > > >> > > > > and > > > > > > > > >> > > > > >> > most > > > > > > > > >> > > > > >> > > > > > clients will still likely timeout > after > > > 30 > > > > > > > seconds. > > > > > > > > >> Then > > > > > > > > >> > > it > > > > > > > > >> > > > is > > > > > > > > >> > > > > >> not > > > > > > > > >> > > > > >> > > > > > obviously what is the benefit to > client > > > > since > > > > > > > > client > > > > > > > > >> > will > > > > > > > > >> > > > > >> timeout > > > > > > > > >> > > > > >> > > after > > > > > > > > >> > > > > >> > > > > 30 > > > > > > > > >> > > > > >> > > > > > seconds before possibly re-connecting > > to > > > > > > broker1, > > > > > > > > >> with > > > > > > > > >> > or > > > > > > > > >> > > > > >> without > > > > > > > > >> > > > > >> > > > > KIP-291. > > > > > > > > >> > > > > >> > > > > > Did I miss something here? > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > 2) I guess Eno's is asking for the > > > specific > > > > > > > > benefits > > > > > > > > >> of > > > > > > > > >> > > this > > > > > > > > >> > > > > >> KIP to > > > > > > > > >> > > > > >> > > > user > > > > > > > > >> > > > > >> > > > > or > > > > > > > > >> > > > > >> > > > > > system administrator, e.g. whether > this > > > KIP > > > > > > > > decreases > > > > > > > > >> > > > average > > > > > > > > >> > > > > >> > > latency, > > > > > > > > >> > > > > >> > > > > > 999th percentile latency, probably of > > > > > exception > > > > > > > > >> exposed > > > > > > > > >> > to > > > > > > > > >> > > > > >> client > > > > > > > > >> > > > > >> > > etc. > > > > > > > > >> > > > > >> > > > It > > > > > > > > >> > > > > >> > > > > > is probably useful to clarify this. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > 3) Does this KIP help improve user > > > > experience > > > > > > > only > > > > > > > > >> when > > > > > > > > >> > > > there > > > > > > > > >> > > > > is > > > > > > > > >> > > > > >> > > issue > > > > > > > > >> > > > > >> > > > > with > > > > > > > > >> > > > > >> > > > > > broker, e.g. significant backlog in > the > > > > > request > > > > > > > > queue > > > > > > > > >> > due > > > > > > > > >> > > to > > > > > > > > >> > > > > >> slow > > > > > > > > >> > > > > >> > > disk > > > > > > > > >> > > > > >> > > > as > > > > > > > > >> > > > > >> > > > > > described in the Google doc? Or is > this > > > KIP > > > > > > also > > > > > > > > >> useful > > > > > > > > >> > > when > > > > > > > > >> > > > > >> there > > > > > > > > >> > > > > >> > is > > > > > > > > >> > > > > >> > > > no > > > > > > > > >> > > > > >> > > > > > ongoing issue in the cluster? It > might > > be > > > > > > helpful > > > > > > > > to > > > > > > > > >> > > clarify > > > > > > > > >> > > > > >> this > > > > > > > > >> > > > > >> > to > > > > > > > > >> > > > > >> > > > > > understand the benefit of this KIP. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > Thanks much, > > > > > > > > >> > > > > >> > > > > > Dong > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > On Fri, Jun 29, 2018 at 2:58 PM, > Lucas > > > > Wang < > > > > > > > > >> > > > > >> lucasatu...@gmail.com > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > Hi Eno, > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > Sorry for the delay in getting the > > > > > experiment > > > > > > > > >> results. > > > > > > > > >> > > > > >> > > > > > > Here is a link to the positive > impact > > > > > > achieved > > > > > > > by > > > > > > > > >> > > > > implementing > > > > > > > > >> > > > > >> > the > > > > > > > > >> > > > > >> > > > > > proposed > > > > > > > > >> > > > > >> > > > > > > change: > > > > > > > > >> > > > > >> > > > > > > https://docs.google.com/ > document/d/ > > > > > > > > >> > > > > 1ge2jjp5aPTBber6zaIT9AdhW > > > > > > > > >> > > > > >> > > > > > > FWUENJ3JO6Zyu4f9tgQ/edit?usp= > sharing > > > > > > > > >> > > > > >> > > > > > > Please take a look when you have > time > > > and > > > > > let > > > > > > > me > > > > > > > > >> know > > > > > > > > >> > > your > > > > > > > > >> > > > > >> > > feedback. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > Regards, > > > > > > > > >> > > > > >> > > > > > > Lucas > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > On Tue, Jun 26, 2018 at 9:52 AM, > > > Harsha < > > > > > > > > >> > > ka...@harsha.io> > > > > > > > > >> > > > > >> wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > Thanks for the pointer. Will > take a > > > > look > > > > > > > might > > > > > > > > >> suit > > > > > > > > >> > > our > > > > > > > > >> > > > > >> > > > requirements > > > > > > > > >> > > > > >> > > > > > > > better. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > Thanks, > > > > > > > > >> > > > > >> > > > > > > > Harsha > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > On Mon, Jun 25th, 2018 at 2:52 > PM, > > > > Lucas > > > > > > > Wang < > > > > > > > > >> > > > > >> > > > lucasatu...@gmail.com > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > Hi Harsha, > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > If I understand correctly, the > > > > > > replication > > > > > > > > >> quota > > > > > > > > >> > > > > mechanism > > > > > > > > >> > > > > >> > > > proposed > > > > > > > > >> > > > > >> > > > > > in > > > > > > > > >> > > > > >> > > > > > > > > KIP-73 can be helpful in that > > > > scenario. > > > > > > > > >> > > > > >> > > > > > > > > Have you tried it out? > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > Thanks, > > > > > > > > >> > > > > >> > > > > > > > > Lucas > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > On Sun, Jun 24, 2018 at 8:28 > AM, > > > > > Harsha < > > > > > > > > >> > > > > ka...@harsha.io > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > Hi Lucas, > > > > > > > > >> > > > > >> > > > > > > > > > One more question, any > thoughts > > > on > > > > > > making > > > > > > > > >> this > > > > > > > > >> > > > > >> configurable > > > > > > > > >> > > > > >> > > > > > > > > > and also allowing subset of > > data > > > > > > requests > > > > > > > > to > > > > > > > > >> be > > > > > > > > >> > > > > >> > prioritized. > > > > > > > > >> > > > > >> > > > For > > > > > > > > >> > > > > >> > > > > > > > example > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > ,we notice in our cluster > when > > we > > > > > take > > > > > > > out > > > > > > > > a > > > > > > > > >> > > broker > > > > > > > > >> > > > > and > > > > > > > > >> > > > > >> > bring > > > > > > > > >> > > > > >> > > > new > > > > > > > > >> > > > > >> > > > > > one > > > > > > > > >> > > > > >> > > > > > > > it > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > will try to become follower > and > > > > have > > > > > > lot > > > > > > > of > > > > > > > > >> > fetch > > > > > > > > >> > > > > >> requests > > > > > > > > >> > > > > >> > to > > > > > > > > >> > > > > >> > > > > other > > > > > > > > >> > > > > >> > > > > > > > > leaders > > > > > > > > >> > > > > >> > > > > > > > > > in clusters. This will > > negatively > > > > > > effect > > > > > > > > the > > > > > > > > >> > > > > >> > > application/client > > > > > > > > >> > > > > >> > > > > > > > > requests. > > > > > > > > >> > > > > >> > > > > > > > > > We are also exploring the > > similar > > > > > > > solution > > > > > > > > to > > > > > > > > >> > > > > >> de-prioritize > > > > > > > > >> > > > > >> > > if > > > > > > > > >> > > > > >> > > > a > > > > > > > > >> > > > > >> > > > > > new > > > > > > > > >> > > > > >> > > > > > > > > > replica comes in for fetch > > > > requests, > > > > > we > > > > > > > are > > > > > > > > >> ok > > > > > > > > >> > > with > > > > > > > > >> > > > > the > > > > > > > > >> > > > > >> > > replica > > > > > > > > >> > > > > >> > > > > to > > > > > > > > >> > > > > >> > > > > > be > > > > > > > > >> > > > > >> > > > > > > > > > taking time but the leaders > > > should > > > > > > > > prioritize > > > > > > > > >> > the > > > > > > > > >> > > > > client > > > > > > > > >> > > > > >> > > > > requests. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > Thanks, > > > > > > > > >> > > > > >> > > > > > > > > > Harsha > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > On Fri, Jun 22nd, 2018 at > 11:35 > > > AM > > > > > > Lucas > > > > > > > > Wang > > > > > > > > >> > > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > Hi Eno, > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > Sorry for the delayed > > response. > > > > > > > > >> > > > > >> > > > > > > > > > > - I haven't implemented the > > > > feature > > > > > > > yet, > > > > > > > > >> so no > > > > > > > > >> > > > > >> > experimental > > > > > > > > >> > > > > >> > > > > > results > > > > > > > > >> > > > > >> > > > > > > > so > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > far. > > > > > > > > >> > > > > >> > > > > > > > > > > And I plan to test in out > in > > > the > > > > > > > > following > > > > > > > > >> > days. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > - You are absolutely right > > that > > > > the > > > > > > > > >> priority > > > > > > > > >> > > queue > > > > > > > > >> > > > > >> does > > > > > > > > >> > > > > >> > not > > > > > > > > >> > > > > >> > > > > > > > completely > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > prevent > > > > > > > > >> > > > > >> > > > > > > > > > > data requests being > processed > > > > ahead > > > > > > of > > > > > > > > >> > > controller > > > > > > > > >> > > > > >> > requests. > > > > > > > > >> > > > > >> > > > > > > > > > > That being said, I expect > it > > to > > > > > > greatly > > > > > > > > >> > mitigate > > > > > > > > >> > > > the > > > > > > > > >> > > > > >> > effect > > > > > > > > >> > > > > >> > > > of > > > > > > > > >> > > > > >> > > > > > > stable > > > > > > > > >> > > > > >> > > > > > > > > > > metadata. > > > > > > > > >> > > > > >> > > > > > > > > > > In any case, I'll try it > out > > > and > > > > > post > > > > > > > the > > > > > > > > >> > > results > > > > > > > > >> > > > > >> when I > > > > > > > > >> > > > > >> > > have > > > > > > > > >> > > > > >> > > > > it. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > Regards, > > > > > > > > >> > > > > >> > > > > > > > > > > Lucas > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > On Wed, Jun 20, 2018 at > 5:44 > > > AM, > > > > > Eno > > > > > > > > >> Thereska > > > > > > > > >> > < > > > > > > > > >> > > > > >> > > > > > > > eno.there...@gmail.com > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Hi Lucas, > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Sorry for the delay, just > > > had a > > > > > > look > > > > > > > at > > > > > > > > >> > this. > > > > > > > > >> > > A > > > > > > > > >> > > > > >> couple > > > > > > > > >> > > > > >> > of > > > > > > > > >> > > > > >> > > > > > > > questions: > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > - did you notice any > > positive > > > > > > change > > > > > > > > >> after > > > > > > > > >> > > > > >> implementing > > > > > > > > >> > > > > >> > > > this > > > > > > > > >> > > > > >> > > > > > KIP? > > > > > > > > >> > > > > >> > > > > > > > > I'm > > > > > > > > >> > > > > >> > > > > > > > > > > > wondering if you have any > > > > > > > experimental > > > > > > > > >> > results > > > > > > > > >> > > > > that > > > > > > > > >> > > > > >> > show > > > > > > > > >> > > > > >> > > > the > > > > > > > > >> > > > > >> > > > > > > > benefit > > > > > > > > >> > > > > >> > > > > > > > > of > > > > > > > > >> > > > > >> > > > > > > > > > > the > > > > > > > > >> > > > > >> > > > > > > > > > > > two queues. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > - priority is usually not > > > > > > sufficient > > > > > > > in > > > > > > > > >> > > > addressing > > > > > > > > >> > > > > >> the > > > > > > > > >> > > > > >> > > > > problem > > > > > > > > >> > > > > >> > > > > > > the > > > > > > > > >> > > > > >> > > > > > > > > KIP > > > > > > > > >> > > > > >> > > > > > > > > > > > identifies. Even with > > > priority > > > > > > > queues, > > > > > > > > >> you > > > > > > > > >> > > will > > > > > > > > >> > > > > >> > sometimes > > > > > > > > >> > > > > >> > > > > > > (often?) > > > > > > > > >> > > > > >> > > > > > > > > have > > > > > > > > >> > > > > >> > > > > > > > > > > the > > > > > > > > >> > > > > >> > > > > > > > > > > > case that data plane > > requests > > > > > will > > > > > > be > > > > > > > > >> ahead > > > > > > > > >> > of > > > > > > > > >> > > > the > > > > > > > > >> > > > > >> > > control > > > > > > > > >> > > > > >> > > > > > plane > > > > > > > > >> > > > > >> > > > > > > > > > > requests. > > > > > > > > >> > > > > >> > > > > > > > > > > > This happens because the > > > system > > > > > > might > > > > > > > > >> have > > > > > > > > >> > > > already > > > > > > > > >> > > > > >> > > started > > > > > > > > >> > > > > >> > > > > > > > > processing > > > > > > > > >> > > > > >> > > > > > > > > > > the > > > > > > > > >> > > > > >> > > > > > > > > > > > data plane requests > before > > > the > > > > > > > control > > > > > > > > >> plane > > > > > > > > >> > > > ones > > > > > > > > >> > > > > >> > > arrived. > > > > > > > > >> > > > > >> > > > So > > > > > > > > >> > > > > >> > > > > > it > > > > > > > > >> > > > > >> > > > > > > > > would > > > > > > > > >> > > > > >> > > > > > > > > > > be > > > > > > > > >> > > > > >> > > > > > > > > > > > good to know what % of > the > > > > > problem > > > > > > > this > > > > > > > > >> KIP > > > > > > > > >> > > > > >> addresses. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Thanks > > > > > > > > >> > > > > >> > > > > > > > > > > > Eno > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > On Fri, Jun 15, 2018 at > > 4:44 > > > > PM, > > > > > > Ted > > > > > > > > Yu < > > > > > > > > >> > > > > >> > > > > yuzhih...@gmail.com > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > Change looks good. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > Thanks > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > On Fri, Jun 15, 2018 at > > > 8:42 > > > > > AM, > > > > > > > > Lucas > > > > > > > > >> > Wang > > > > > > > > >> > > < > > > > > > > > >> > > > > >> > > > > > > > lucasatu...@gmail.com > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Hi Ted, > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Thanks for the > > > suggestion. > > > > > I've > > > > > > > > >> updated > > > > > > > > >> > > the > > > > > > > > >> > > > > KIP. > > > > > > > > >> > > > > >> > > Please > > > > > > > > >> > > > > >> > > > > > take > > > > > > > > >> > > > > >> > > > > > > > > > another > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > look. > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Lucas > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > On Thu, Jun 14, 2018 > at > > > > 6:34 > > > > > > PM, > > > > > > > > Ted > > > > > > > > >> Yu > > > > > > > > >> > < > > > > > > > > >> > > > > >> > > > > > > yuzhih...@gmail.com > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > wrote: > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > Currently in > > > > > > KafkaConfig.scala > > > > > > > : > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > val > > QueuedMaxRequests = > > > > 500 > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > It would be good if > > you > > > > can > > > > > > > > include > > > > > > > > >> > the > > > > > > > > >> > > > > >> default > > > > > > > > >> > > > > >> > > value > > > > > > > > >> > > > > >> > > > > for > > > > > > > > >> > > > > >> > > > > > > > this > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > > >
