Hi Dong,
The simulation of disk being slow is merely for me to easily construct a
testing scenario
with a backlog of produce requests. In production, other than the disk
being slow, a backlog of
produce requests may also be caused by high produce QPS.
In that case, we may not want to kill the broker and that's when this KIP
can be useful, both for JBOD
and non-JBOD setup.
Going back to your previous question about each ProduceRequest covering 20
partitions that are randomly
distributed, let's say a LeaderAndIsr request is enqueued that tries to
switch the current broker, say broker0, from leader to follower
*for one of the partitions*, say *test-0*. For the sake of argument, let's
also assume the other brokers, say broker1, have *stopped* fetching from
the current broker, i.e. broker0.
1. If the enqueued produce requests have acks = -1 (ALL)
1.1 without this KIP, the ProduceRequests ahead of LeaderAndISR will be
put into the purgatory,
and since they'll never be replicated to other brokers (because of
the assumption made above), they will
be completed either when the LeaderAndISR request is processed or
when the timeout happens.
1.2 With this KIP, broker0 will immediately transition the partition
test-0 to become a follower,
after the current broker sees the replication of the remaining 19
partitions, it can send a response indicating that
it's no longer the leader for the "test-0".
To see the latency difference between 1.1 and 1.2, let's say there are
24K produce requests ahead of the LeaderAndISR, and there are 8 io threads,
so each io thread will process approximately 3000 produce requests. Now
let's investigate the io thread that finally processed the LeaderAndISR.
For the 3000 produce requests, if we model the time when their remaining
19 partitions catch up as t0, t1, ...t2999, and the LeaderAndISR request is
processed at time t3000.
Without this KIP, the 1st produce request would have waited an extra
t3000 - t0 time in the purgatory, the 2nd an extra time of t3000 - t1, etc.
Roughly speaking, the latency difference is bigger for the earlier
produce requests than for the later ones. For the same reason, the more
ProduceRequests queued
before the LeaderAndISR, the bigger benefit we get (capped by the produce
timeout).
2. If the enqueued produce requests have acks=0 or acks=1
There will be no latency differences in this case, but
2.1 without this KIP, the records of partition test-0 in the
ProduceRequests ahead of the LeaderAndISR will be appended to the local log,
and eventually be truncated after processing the LeaderAndISR. This
is what's referred to as
"some unofficial definition of data loss in terms of messages
beyond the high watermark".
2.2 with this KIP, we can mitigate the effect since if the LeaderAndISR
is immediately processed, the response to producers will have
the NotLeaderForPartition error, causing producers to retry
This explanation above is the benefit for reducing the latency of a broker
becoming the follower,
closely related is reducing the latency of a broker becoming the leader.
In this case, the benefit is even more obvious, if other brokers have
resigned leadership, and the
current broker should take leadership. Any delay in processing the
LeaderAndISR will be perceived
by clients as unavailability. In extreme cases, this can cause failed
produce requests if the retries are
exhausted.
Another two types of controller requests are UpdateMetadata and
StopReplica, which I'll briefly discuss as follows:
For UpdateMetadata requests, delayed processing means clients receiving
stale metadata, e.g. with the wrong leadership info
for certain partitions, and the effect is more retries or even fatal
failure if the retries are exhausted.
For StopReplica requests, a long queuing time may degrade the performance
of topic deletion.
Regarding your last question of the delay for DescribeLogDirsRequest, you
are right
that this KIP cannot help with the latency in getting the log dirs info,
and it's only relevant
when controller requests are involved.
Regards,
Lucas
On Tue, Jul 3, 2018 at 5:11 PM, Dong Lin <lindon...@gmail.com> wrote:
> Hey Jun,
>
> Thanks much for the comments. It is good point. So the feature may be
> useful for JBOD use-case. I have one question below.
>
> Hey Lucas,
>
> Do you think this feature is also useful for non-JBOD setup or it is only
> useful for the JBOD setup? It may be useful to understand this.
>
> When the broker is setup using JBOD, in order to move leaders on the failed
> disk to other disks, the system operator first needs to get the list of
> partitions on the failed disk. This is currently achieved using
> AdminClient.describeLogDirs(), which sends DescribeLogDirsRequest to the
> broker. If we only prioritize the controller requests, then the
> DescribeLogDirsRequest
> may still take a long time to be processed by the broker. So the overall
> time to move leaders away from the failed disk may still be long even with
> this KIP. What do you think?
>
> Thanks,
> Dong
>
>
> On Tue, Jul 3, 2018 at 4:38 PM, Lucas Wang <lucasatu...@gmail.com> wrote:
>
> > Thanks for the insightful comment, Jun.
> >
> > @Dong,
> > Since both of the two comments in your previous email are about the
> > benefits of this KIP and whether it's useful,
> > in light of Jun's last comment, do you agree that this KIP can be
> > beneficial in the case mentioned by Jun?
> > Please let me know, thanks!
> >
> > Regards,
> > Lucas
> >
> > On Tue, Jul 3, 2018 at 2:07 PM, Jun Rao <j...@confluent.io> wrote:
> >
> > > Hi, Lucas, Dong,
> > >
> > > If all disks on a broker are slow, one probably should just kill the
> > > broker. In that case, this KIP may not help. If only one of the disks
> on
> > a
> > > broker is slow, one may want to fail that disk and move the leaders on
> > that
> > > disk to other brokers. In that case, being able to process the
> > LeaderAndIsr
> > > requests faster will potentially help the producers recover quicker.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jul 2, 2018 at 7:56 PM, Dong Lin <lindon...@gmail.com> wrote:
> > >
> > > > Hey Lucas,
> > > >
> > > > Thanks for the reply. Some follow up questions below.
> > > >
> > > > Regarding 1, if each ProduceRequest covers 20 partitions that are
> > > randomly
> > > > distributed across all partitions, then each ProduceRequest will
> likely
> > > > cover some partitions for which the broker is still leader after it
> > > quickly
> > > > processes the
> > > > LeaderAndIsrRequest. Then broker will still be slow in processing
> these
> > > > ProduceRequest and request will still be very high with this KIP. It
> > > seems
> > > > that most ProduceRequest will still timeout after 30 seconds. Is this
> > > > understanding correct?
> > > >
> > > > Regarding 2, if most ProduceRequest will still timeout after 30
> > seconds,
> > > > then it is less clear how this KIP reduces average produce latency.
> Can
> > > you
> > > > clarify what metrics can be improved by this KIP?
> > > >
> > > > Not sure why system operator directly cares number of truncated
> > messages.
> > > > Do you mean this KIP can improve average throughput or reduce message
> > > > duplication? It will be good to understand this.
> > > >
> > > > Thanks,
> > > > Dong
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, 3 Jul 2018 at 7:12 AM Lucas Wang <lucasatu...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Dong,
> > > > >
> > > > > Thanks for your valuable comments. Please see my reply below.
> > > > >
> > > > > 1. The Google doc showed only 1 partition. Now let's consider a
> more
> > > > common
> > > > > scenario
> > > > > where broker0 is the leader of many partitions. And let's say for
> > some
> > > > > reason its IO becomes slow.
> > > > > The number of leader partitions on broker0 is so large, say 10K,
> that
> > > the
> > > > > cluster is skewed,
> > > > > and the operator would like to shift the leadership for a lot of
> > > > > partitions, say 9K, to other brokers,
> > > > > either manually or through some service like cruise control.
> > > > > With this KIP, not only will the leadership transitions finish more
> > > > > quickly, helping the cluster itself becoming more balanced,
> > > > > but all existing producers corresponding to the 9K partitions will
> > get
> > > > the
> > > > > errors relatively quickly
> > > > > rather than relying on their timeout, thanks to the batched async
> ZK
> > > > > operations.
> > > > > To me it's a useful feature to have during such troublesome times.
> > > > >
> > > > >
> > > > > 2. The experiments in the Google Doc have shown that with this KIP
> > many
> > > > > producers
> > > > > receive an explicit error NotLeaderForPartition, based on which
> they
> > > > retry
> > > > > immediately.
> > > > > Therefore the latency (~14 seconds+quick retry) for their single
> > > message
> > > > is
> > > > > much smaller
> > > > > compared with the case of timing out without the KIP (30 seconds
> for
> > > > timing
> > > > > out + quick retry).
> > > > > One might argue that reducing the timing out on the producer side
> can
> > > > > achieve the same result,
> > > > > yet reducing the timeout has its own drawbacks[1].
> > > > >
> > > > > Also *IF* there were a metric to show the number of truncated
> > messages
> > > on
> > > > > brokers,
> > > > > with the experiments done in the Google Doc, it should be easy to
> see
> > > > that
> > > > > a lot fewer messages need
> > > > > to be truncated on broker0 since the up-to-date metadata avoids
> > > appending
> > > > > of messages
> > > > > in subsequent PRODUCE requests. If we talk to a system operator and
> > ask
> > > > > whether
> > > > > they prefer fewer wasteful IOs, I bet most likely the answer is
> yes.
> > > > >
> > > > > 3. To answer your question, I think it might be helpful to
> construct
> > > some
> > > > > formulas.
> > > > > To simplify the modeling, I'm going back to the case where there is
> > > only
> > > > > ONE partition involved.
> > > > > Following the experiments in the Google Doc, let's say broker0
> > becomes
> > > > the
> > > > > follower at time t0,
> > > > > and after t0 there were still N produce requests in its request
> > queue.
> > > > > With the up-to-date metadata brought by this KIP, broker0 can reply
> > > with
> > > > an
> > > > > NotLeaderForPartition exception,
> > > > > let's use M1 to denote the average processing time of replying with
> > > such
> > > > an
> > > > > error message.
> > > > > Without this KIP, the broker will need to append messages to
> > segments,
> > > > > which may trigger a flush to disk,
> > > > > let's use M2 to denote the average processing time for such logic.
> > > > > Then the average extra latency incurred without this KIP is N *
> (M2 -
> > > > M1) /
> > > > > 2.
> > > > >
> > > > > In practice, M2 should always be larger than M1, which means as
> long
> > > as N
> > > > > is positive,
> > > > > we would see improvements on the average latency.
> > > > > There does not need to be significant backlog of requests in the
> > > request
> > > > > queue,
> > > > > or severe degradation of disk performance to have the improvement.
> > > > >
> > > > > Regards,
> > > > > Lucas
> > > > >
> > > > >
> > > > > [1] For instance, reducing the timeout on the producer side can
> > trigger
> > > > > unnecessary duplicate requests
> > > > > when the corresponding leader broker is overloaded, exacerbating
> the
> > > > > situation.
> > > > >
> > > > > On Sun, Jul 1, 2018 at 9:18 PM, Dong Lin <lindon...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hey Lucas,
> > > > > >
> > > > > > Thanks much for the detailed documentation of the experiment.
> > > > > >
> > > > > > Initially I also think having a separate queue for controller
> > > requests
> > > > is
> > > > > > useful because, as you mentioned in the summary section of the
> > Google
> > > > > doc,
> > > > > > controller requests are generally more important than data
> requests
> > > and
> > > > > we
> > > > > > probably want controller requests to be processed sooner. But
> then
> > > Eno
> > > > > has
> > > > > > two very good questions which I am not sure the Google doc has
> > > answered
> > > > > > explicitly. Could you help with the following questions?
> > > > > >
> > > > > > 1) It is not very clear what is the actual benefit of KIP-291 to
> > > users.
> > > > > The
> > > > > > experiment setup in the Google doc simulates the scenario that
> > broker
> > > > is
> > > > > > very slow handling ProduceRequest due to e.g. slow disk. It
> > currently
> > > > > > assumes that there is only 1 partition. But in the common
> scenario,
> > > it
> > > > is
> > > > > > probably reasonable to assume that there are many other
> partitions
> > > that
> > > > > are
> > > > > > also actively produced to and ProduceRequest to these partition
> > also
> > > > > takes
> > > > > > e.g. 2 seconds to be processed. So even if broker0 can become
> > > follower
> > > > > for
> > > > > > the partition 0 soon, it probably still needs to process the
> > > > > ProduceRequest
> > > > > > slowly t in the queue because these ProduceRequests cover other
> > > > > partitions.
> > > > > > Thus most ProduceRequest will still timeout after 30 seconds and
> > most
> > > > > > clients will still likely timeout after 30 seconds. Then it is
> not
> > > > > > obviously what is the benefit to client since client will timeout
> > > after
> > > > > 30
> > > > > > seconds before possibly re-connecting to broker1, with or without
> > > > > KIP-291.
> > > > > > Did I miss something here?
> > > > > >
> > > > > > 2) I guess Eno's is asking for the specific benefits of this KIP
> to
> > > > user
> > > > > or
> > > > > > system administrator, e.g. whether this KIP decreases average
> > > latency,
> > > > > > 999th percentile latency, probably of exception exposed to client
> > > etc.
> > > > It
> > > > > > is probably useful to clarify this.
> > > > > >
> > > > > > 3) Does this KIP help improve user experience only when there is
> > > issue
> > > > > with
> > > > > > broker, e.g. significant backlog in the request queue due to slow
> > > disk
> > > > as
> > > > > > described in the Google doc? Or is this KIP also useful when
> there
> > is
> > > > no
> > > > > > ongoing issue in the cluster? It might be helpful to clarify this
> > to
> > > > > > understand the benefit of this KIP.
> > > > > >
> > > > > >
> > > > > > Thanks much,
> > > > > > Dong
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 29, 2018 at 2:58 PM, Lucas Wang <
> lucasatu...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Eno,
> > > > > > >
> > > > > > > Sorry for the delay in getting the experiment results.
> > > > > > > Here is a link to the positive impact achieved by implementing
> > the
> > > > > > proposed
> > > > > > > change:
> > > > > > > https://docs.google.com/document/d/1ge2jjp5aPTBber6zaIT9AdhW
> > > > > > > FWUENJ3JO6Zyu4f9tgQ/edit?usp=sharing
> > > > > > > Please take a look when you have time and let me know your
> > > feedback.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Lucas
> > > > > > >
> > > > > > > On Tue, Jun 26, 2018 at 9:52 AM, Harsha <ka...@harsha.io>
> wrote:
> > > > > > >
> > > > > > > > Thanks for the pointer. Will take a look might suit our
> > > > requirements
> > > > > > > > better.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Harsha
> > > > > > > >
> > > > > > > > On Mon, Jun 25th, 2018 at 2:52 PM, Lucas Wang <
> > > > lucasatu...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Harsha,
> > > > > > > > >
> > > > > > > > > If I understand correctly, the replication quota mechanism
> > > > proposed
> > > > > > in
> > > > > > > > > KIP-73 can be helpful in that scenario.
> > > > > > > > > Have you tried it out?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Lucas
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Jun 24, 2018 at 8:28 AM, Harsha < ka...@harsha.io
> >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Lucas,
> > > > > > > > > > One more question, any thoughts on making this
> configurable
> > > > > > > > > > and also allowing subset of data requests to be
> > prioritized.
> > > > For
> > > > > > > > example
> > > > > > > > >
> > > > > > > > > > ,we notice in our cluster when we take out a broker and
> > bring
> > > > new
> > > > > > one
> > > > > > > > it
> > > > > > > > >
> > > > > > > > > > will try to become follower and have lot of fetch
> requests
> > to
> > > > > other
> > > > > > > > > leaders
> > > > > > > > > > in clusters. This will negatively effect the
> > > application/client
> > > > > > > > > requests.
> > > > > > > > > > We are also exploring the similar solution to
> de-prioritize
> > > if
> > > > a
> > > > > > new
> > > > > > > > > > replica comes in for fetch requests, we are ok with the
> > > replica
> > > > > to
> > > > > > be
> > > > > > > > > > taking time but the leaders should prioritize the client
> > > > > requests.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Harsha
> > > > > > > > > >
> > > > > > > > > > On Fri, Jun 22nd, 2018 at 11:35 AM Lucas Wang wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Eno,
> > > > > > > > > > >
> > > > > > > > > > > Sorry for the delayed response.
> > > > > > > > > > > - I haven't implemented the feature yet, so no
> > experimental
> > > > > > results
> > > > > > > > so
> > > > > > > > >
> > > > > > > > > > > far.
> > > > > > > > > > > And I plan to test in out in the following days.
> > > > > > > > > > >
> > > > > > > > > > > - You are absolutely right that the priority queue does
> > not
> > > > > > > > completely
> > > > > > > > >
> > > > > > > > > > > prevent
> > > > > > > > > > > data requests being processed ahead of controller
> > requests.
> > > > > > > > > > > That being said, I expect it to greatly mitigate the
> > effect
> > > > of
> > > > > > > stable
> > > > > > > > > > > metadata.
> > > > > > > > > > > In any case, I'll try it out and post the results when
> I
> > > have
> > > > > it.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Lucas
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jun 20, 2018 at 5:44 AM, Eno Thereska <
> > > > > > > > eno.there...@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Lucas,
> > > > > > > > > > > >
> > > > > > > > > > > > Sorry for the delay, just had a look at this. A
> couple
> > of
> > > > > > > > questions:
> > > > > > > > >
> > > > > > > > > > > > - did you notice any positive change after
> implementing
> > > > this
> > > > > > KIP?
> > > > > > > > > I'm
> > > > > > > > > > > > wondering if you have any experimental results that
> > show
> > > > the
> > > > > > > > benefit
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > two queues.
> > > > > > > > > > > >
> > > > > > > > > > > > - priority is usually not sufficient in addressing
> the
> > > > > problem
> > > > > > > the
> > > > > > > > > KIP
> > > > > > > > > > > > identifies. Even with priority queues, you will
> > sometimes
> > > > > > > (often?)
> > > > > > > > > have
> > > > > > > > > > > the
> > > > > > > > > > > > case that data plane requests will be ahead of the
> > > control
> > > > > > plane
> > > > > > > > > > > requests.
> > > > > > > > > > > > This happens because the system might have already
> > > started
> > > > > > > > > processing
> > > > > > > > > > > the
> > > > > > > > > > > > data plane requests before the control plane ones
> > > arrived.
> > > > So
> > > > > > it
> > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > > good to know what % of the problem this KIP
> addresses.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > > Eno
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jun 15, 2018 at 4:44 PM, Ted Yu <
> > > > > yuzhih...@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Change looks good.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jun 15, 2018 at 8:42 AM, Lucas Wang <
> > > > > > > > lucasatu...@gmail.com
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the suggestion. I've updated the KIP.
> > > Please
> > > > > > take
> > > > > > > > > > another
> > > > > > > > > > >
> > > > > > > > > > > > > look.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Lucas
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Jun 14, 2018 at 6:34 PM, Ted Yu <
> > > > > > > yuzhih...@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Currently in KafkaConfig.scala :
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > val QueuedMaxRequests = 500
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It would be good if you can include the default
> > > value
> > > > > for
> > > > > > > > this
> > > > > > > > >
> > > > > > > > > > new
> > > > > > > > > > >
> > > > > > > > > > > > > config
> > > > > > > > > > > > > > > in the KIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Jun 14, 2018 at 4:28 PM, Lucas Wang <
> > > > > > > > > > lucasatu...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Ted, Dong
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I've updated the KIP by adding a new config,
> > > > instead
> > > > > of
> > > > > > > > > reusing
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > existing one.
> > > > > > > > > > > > > > > > Please take another look when you have time.
> > > > Thanks a
> > > > > > > lot!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Lucas
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Jun 14, 2018 at 2:33 PM, Ted Yu <
> > > > > > > > yuzhih...@gmail.com
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > bq. that's a waste of resource if control
> > > request
> > > > > > rate
> > > > > > > is
> > > > > > > > > low
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I don't know if control request rate can
> get
> > to
> > > > > > > 100,000,
> > > > > > > > > > > likely
> > > > > > > > > > > > > not.
> > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > using the same bound as that for data
> > requests
> > > > > seems
> > > > > > > > high.
> > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Jun 13, 2018 at 10:13 PM, Lucas
> Wang
> > <
> > > > > > > > > > > > > lucasatu...@gmail.com >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Ted,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for taking a look at this KIP.
> > > > > > > > > > > > > > > > > > Let's say today the setting of
> > > > > > "queued.max.requests"
> > > > > > > in
> > > > > > > > > > > > cluster A
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > 1000,
> > > > > > > > > > > > > > > > > > while the setting in cluster B is
> 100,000.
> > > > > > > > > > > > > > > > > > The 100 times difference might have
> > indicated
> > > > > that
> > > > > > > > > machines
> > > > > > > > > > > in
> > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > B
> > > > > > > > > > > > > > > > > > have larger memory.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > By reusing the "queued.max.requests", the
> > > > > > > > > > > controlRequestQueue
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > cluster
> > > > > > > > > > > > > > > > > B
> > > > > > > > > > > > > > > > > > automatically
> > > > > > > > > > > > > > > > > > gets a 100x capacity without explicitly
> > > > bothering
> > > > > > the
> > > > > > > > > > > > operators.
> > > > > > > > > > > > > > > > > > I understand the counter argument can be
> > that
> > > > > maybe
> > > > > > > > > that's
> > > > > > > > > > a
> > > > > > > > > > >
> > > > > > > > > > > > > waste
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > resource if control request
> > > > > > > > > > > > > > > > > > rate is low and operators may want to
> fine
> > > tune
> > > > > the
> > > > > > > > > > capacity
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > controlRequestQueue.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'm ok with either approach, and can
> change
> > > it
> > > > if
> > > > > > you
> > > > > > > > or
> > > > > > > > >
> > > > > > > > > > > anyone
> > > > > > > > > > > > > > else
> > > > > > > > > > > > > > > > > feels
> > > > > > > > > > > > > > > > > > strong about adding the extra config.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Lucas
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Wed, Jun 13, 2018 at 3:11 PM, Ted Yu <
> > > > > > > > > > yuzhih...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Lucas:
> > > > > > > > > > > > > > > > > > > Under Rejected Alternatives, #2, can
> you
> > > > > > elaborate
> > > > > > > a
> > > > > > > > > bit
> > > > > > > > > > > more
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > why
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > separate config has bigger impact ?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Jun 13, 2018 at 2:00 PM, Dong
> > Lin <
> > > > > > > > > > > > lindon...@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hey Luca,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the KIP. Looks good
> overall.
> > > > Some
> > > > > > > > > comments
> > > > > > > > > > > > below:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > - We usually specify the full mbean
> for
> > > the
> > > > > new
> > > > > > > > > metrics
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > specify it in the Public Interface
> > > section
> > > > > > > similar
> > > > > > > > > to
> > > > > > > > > > > > KIP-237
> > > > > > > > > > > > > > > > > > > > < https://cwiki.apache.org/
> > > > > > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 237%3A+More+Controller+Health+
> Metrics>
> > > > > > > > > > > > > > > > > > > > ?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > - Maybe we could follow the same
> > pattern
> > > as
> > > > > > > KIP-153
> > > > > > > > > > > > > > > > > > > > < https://cwiki.apache.org/
> > > > > > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > 153%3A+Include+only+client+traffic+in+BytesOutPerSec+
> > > > > > > > > > > > > metric>,
> > > > > > > > > > > > > > > > > > > > where we keep the existing sensor
> name
> > > > > > > > > "BytesInPerSec"
> > > > > > > > > > > and
> > > > > > > > > > > > > add
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > sensor
> > > > > > > > > > > > > > > > > > > > "ReplicationBytesInPerSec", rather
> than
> > > > > > replacing
> > > > > > > > > the
> > > > > > > > > > > > sensor
> > > > > > > > > > > > > > > name "
> > > > > > > > > > > > > > > > > > > > BytesInPerSec" with e.g.
> > > > > "ClientBytesInPerSec".
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > - It seems that the KIP changes the
> > > > semantics
> > > > > > of
> > > > > > > > the
> > > > > > > > >
> > > > > > > > > > > broker
> > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > "queued.max.requests" because the
> > number
> > > of
> > > > > > total
> > > > > > > > > > > requests
> > > > > > > > > > > > > > queued
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > broker will be no longer bounded by
> > > > > > > > > > > "queued.max.requests".
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > > needs to be specified in the Public
> > > > > Interfaces
> > > > > > > > > section
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Dong
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Jun 13, 2018 at 12:45 PM,
> Lucas
> > > > Wang
> > > > > <
> > > > > > > > > > > > > > > > lucasatu...@gmail.com >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Kafka experts,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I created KIP-291 to add a separate
> > > queue
> > > > > for
> > > > > > > > > > > controller
> > > > > > > > > > > > > > > > requests:
> > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/
> > > > > > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > >
> > > > > > > > > > > > 291%
> > > > > > > > > > > > > > > > > > > > > 3A+Have+separate+queues+for+
> > > > > > > > > > control+requests+and+data+
> > > > > > > > > > >
> > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Can you please take a look and let
> me
> > > > know
> > > > > > your
> > > > > > > > > > > feedback?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks a lot for your time!
> > > > > > > > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > > > > > > > > Lucas
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
