@Becket Makes sense. I've updated the KIP by adding the following paragraph to the motivation section
> Today there is no separate between controller requests and regular data > plane requests. Specifically (1) a controller in a cluster uses the same > advertised endpoints to connect to brokers as what clients and regular > brokers use for exchanging data (2) on the broker side, the same network > (processor) thread could be multiplexed by handling a controller connection > and many other data plane connections (3) after a controller request is > read from the socket, it is enqueued into the single FIFO requestQueue, > which is used for all types of requests (4) request handler threads poll > requests from the requestQueue and handles the controller requests with the > same priority as regular data requests. > > Because of the multiplexing at every stage of request handling, controller > requests could be significantly delayed under the following scenarios: > > 1. The requestQueue is full, and therefore blocks a network > (processor) thread that has read a controller request from the socket. > 2. A controller request is enqueued into the requestQueue after a > backlog of data requests, and experiences a long queuing time in the > requestQueue. > > Please let me know if that looks ok or any other change you'd like to make. Thanks! Lucas On Mon, Aug 13, 2018 at 6:33 AM, Becket Qin <becket....@gmail.com> wrote: > Hi Lucas, > > Thanks for the explanation. It might be a nitpick, but it seems better to > mention in the motivation part that today the client requests and > controller requests are not only sharing the same queue, but also a bunch > of things else, so that we can avoid asking people to read the rejected > alternatives. > > Thanks, > > Jiangjie (Becket) Qin > > > > > > > > On Fri, Aug 10, 2018 at 6:23 AM, Lucas Wang <lucasatu...@gmail.com> wrote: > > > @Becket, > > > > I've asked for review by Jun and Joel in the vote thread. > > Regarding the separate thread and port, I did talk about it in the > rejected > > alternative design 1. > > Please let me know if you'd like more elaboration or moving it to the > > motivation, etc. > > > > Thanks, > > Lucas > > > > On Wed, Aug 8, 2018 at 3:59 PM, Becket Qin <becket....@gmail.com> wrote: > > > > > Hi Lucas, > > > > > > Yes, a separate Jira is OK. > > > > > > Since the proposal has significantly changed since the initial vote > > > started. We probably should let the others who have already voted know > > and > > > ensure they are happy with the updated proposal. > > > Also, it seems the motivation part of the KIP wiki is still just > talking > > > about the separate queue and not fully cover the changes we make now, > > e.g. > > > separate thread, port, etc. We might want to explain a bit more so for > > > people who did not follow the discussion mail thread also understand > the > > > whole proposal. > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > On Wed, Aug 8, 2018 at 12:44 PM, Lucas Wang <lucasatu...@gmail.com> > > wrote: > > > > > > > Hi Becket, > > > > > > > > Thanks for the review. The current write up in the KIP won’t change > the > > > > ordering behavior. Are you ok with addressing that as a separate > > > > independent issue (I’ll create a separate ticket for it)? > > > > If so, can you please give me a +1 on the vote thread? > > > > > > > > Thanks, > > > > Lucas > > > > > > > > On Tue, Aug 7, 2018 at 7:34 PM Becket Qin <becket....@gmail.com> > > wrote: > > > > > > > > > Thanks for the updated KIP wiki, Lucas. Looks good to me overall. > > > > > > > > > > It might be an implementation detail, but do we still plan to use > the > > > > > correlation id to ensure the request processing order? > > > > > > > > > > Thanks, > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > On Tue, Jul 31, 2018 at 3:39 AM, Lucas Wang <lucasatu...@gmail.com > > > > > > wrote: > > > > > > > > > > > Thanks for your review, Dong. > > > > > > Ack that these configs will have a bigger impact for users. > > > > > > > > > > > > On the other hand, I would argue that the request queue becoming > > full > > > > > > may or may not be a rare scenario. > > > > > > How often the request queue gets full depends on the request > > incoming > > > > > rate, > > > > > > the request processing rate, and the size of the request queue. > > > > > > When that happens, the dedicated endpoints design can better > handle > > > > > > it than any of the previously discussed options. > > > > > > > > > > > > Another reason I made the change was that I have the same taste > > > > > > as Becket that it's a better separation of the control plane from > > the > > > > > data > > > > > > plane. > > > > > > > > > > > > Finally, I want to clarify that this change is NOT motivated by > the > > > > > > out-of-order > > > > > > processing discussion. The latter problem is orthogonal to this > > KIP, > > > > and > > > > > it > > > > > > can happen in any of the design options we discussed for this KIP > > so > > > > far. > > > > > > So I'd like to address out-of-order processing separately in > > another > > > > > > thread, > > > > > > and avoid mentioning it in this KIP. > > > > > > > > > > > > Thanks, > > > > > > Lucas > > > > > > > > > > > > On Fri, Jul 27, 2018 at 7:51 PM, Dong Lin <lindon...@gmail.com> > > > wrote: > > > > > > > > > > > > > Hey Lucas, > > > > > > > > > > > > > > Thanks for the update. > > > > > > > > > > > > > > The current KIP propose new broker configs > > > "listeners.for.controller" > > > > > and > > > > > > > "advertised.listeners.for.controller". This is going to be a > big > > > > change > > > > > > > since listeners are among the most important configs that every > > > user > > > > > > needs > > > > > > > to change. According to the rejected alternative section, it > > seems > > > > that > > > > > > the > > > > > > > reason to add these two configs is to improve performance when > > the > > > > data > > > > > > > request queue is full rather than for correctness. It should > be a > > > > very > > > > > > rare > > > > > > > scenario and I am not sure we should add configs for all users > > just > > > > to > > > > > > > improve the performance in such rare scenario. > > > > > > > > > > > > > > Also, if the new design is based on the issues which are > > discovered > > > > in > > > > > > the > > > > > > > recent discussion, e.g. out of order processing if we don't > use a > > > > > > dedicated > > > > > > > thread for controller request, it may be useful to explain the > > > > problem > > > > > in > > > > > > > the motivation section. > > > > > > > > > > > > > > Thanks, > > > > > > > Dong > > > > > > > > > > > > > > On Fri, Jul 27, 2018 at 1:28 PM, Lucas Wang < > > lucasatu...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > A kind reminder for review of this KIP. > > > > > > > > > > > > > > > > Thank you very much! > > > > > > > > Lucas > > > > > > > > > > > > > > > > On Wed, Jul 25, 2018 at 10:23 PM, Lucas Wang < > > > > lucasatu...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > I've updated the KIP by adding the dedicated endpoints for > > > > > controller > > > > > > > > > connections, > > > > > > > > > and pinning threads for controller requests. > > > > > > > > > Also I've updated the title of this KIP. Please take a look > > and > > > > let > > > > > > me > > > > > > > > > know your feedback. > > > > > > > > > > > > > > > > > > Thanks a lot for your time! > > > > > > > > > Lucas > > > > > > > > > > > > > > > > > > On Tue, Jul 24, 2018 at 10:19 AM, Mayuresh Gharat < > > > > > > > > > gharatmayures...@gmail.com> wrote: > > > > > > > > > > > > > > > > > >> Hi Lucas, > > > > > > > > >> I agree, if we want to go forward with a separate > controller > > > > plane > > > > > > and > > > > > > > > >> data > > > > > > > > >> plane and completely isolate them, having a separate port > > for > > > > > > > controller > > > > > > > > >> with a separate Acceptor and a Processor sounds ideal to > me. > > > > > > > > >> > > > > > > > > >> Thanks, > > > > > > > > >> > > > > > > > > >> Mayuresh > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> On Mon, Jul 23, 2018 at 11:04 PM Becket Qin < > > > > becket....@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >> > Hi Lucas, > > > > > > > > >> > > > > > > > > > >> > Yes, I agree that a dedicated end to end control flow > > would > > > be > > > > > > > ideal. > > > > > > > > >> > > > > > > > > > >> > Thanks, > > > > > > > > >> > > > > > > > > > >> > Jiangjie (Becket) Qin > > > > > > > > >> > > > > > > > > > >> > On Tue, Jul 24, 2018 at 1:05 PM, Lucas Wang < > > > > > > lucasatu...@gmail.com> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > >> > > Thanks for the comment, Becket. > > > > > > > > >> > > So far, we've been trying to avoid making any request > > > > handler > > > > > > > thread > > > > > > > > >> > > special. > > > > > > > > >> > > But if we were to follow that path in order to make > the > > > two > > > > > > planes > > > > > > > > >> more > > > > > > > > >> > > isolated, > > > > > > > > >> > > what do you think about also having a dedicated > > processor > > > > > > thread, > > > > > > > > >> > > and dedicated port for the controller? > > > > > > > > >> > > > > > > > > > > >> > > Today one processor thread can handle multiple > > > connections, > > > > > > let's > > > > > > > > say > > > > > > > > >> 100 > > > > > > > > >> > > connections > > > > > > > > >> > > > > > > > > > > >> > > represented by connection0, ... connection99, among > > which > > > > > > > > >> connection0-98 > > > > > > > > >> > > are from clients, while connection99 is from > > > > > > > > >> > > > > > > > > > > >> > > the controller. Further let's say after one selector > > > > polling, > > > > > > > there > > > > > > > > >> are > > > > > > > > >> > > incoming requests on all connections. > > > > > > > > >> > > > > > > > > > > >> > > When the request queue is full, (either the data > request > > > > being > > > > > > > full > > > > > > > > in > > > > > > > > >> > the > > > > > > > > >> > > two queue design, or > > > > > > > > >> > > > > > > > > > > >> > > the one single queue being full in the deque design), > > the > > > > > > > processor > > > > > > > > >> > thread > > > > > > > > >> > > will be blocked first > > > > > > > > >> > > > > > > > > > > >> > > when trying to enqueue the data request from > > connection0, > > > > then > > > > > > > > >> possibly > > > > > > > > >> > > blocked for the data request > > > > > > > > >> > > > > > > > > > > >> > > from connection1, ... etc even though the controller > > > request > > > > > is > > > > > > > > ready > > > > > > > > >> to > > > > > > > > >> > be > > > > > > > > >> > > enqueued. > > > > > > > > >> > > > > > > > > > > >> > > To solve this problem, it seems we would need to have > a > > > > > separate > > > > > > > > port > > > > > > > > >> > > dedicated to > > > > > > > > >> > > > > > > > > > > >> > > the controller, a dedicated processor thread, a > > dedicated > > > > > > > controller > > > > > > > > >> > > request queue, > > > > > > > > >> > > > > > > > > > > >> > > and pinning of one request handler thread for > controller > > > > > > requests. > > > > > > > > >> > > > > > > > > > > >> > > Thanks, > > > > > > > > >> > > Lucas > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > On Mon, Jul 23, 2018 at 6:00 PM, Becket Qin < > > > > > > becket....@gmail.com > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > >> > > > > > > > > > > >> > > > Personally I am not fond of the dequeue approach > > simply > > > > > > because > > > > > > > it > > > > > > > > >> is > > > > > > > > >> > > > against the basic idea of isolating the controller > > plane > > > > and > > > > > > > data > > > > > > > > >> > plane. > > > > > > > > >> > > > With a single dequeue, theoretically speaking the > > > > controller > > > > > > > > >> requests > > > > > > > > >> > can > > > > > > > > >> > > > starve the clients requests. I would prefer the > > approach > > > > > with > > > > > > a > > > > > > > > >> > separate > > > > > > > > >> > > > controller request queue and a dedicated controller > > > > request > > > > > > > > handler > > > > > > > > >> > > thread. > > > > > > > > >> > > > > > > > > > > > >> > > > Thanks, > > > > > > > > >> > > > > > > > > > > > >> > > > Jiangjie (Becket) Qin > > > > > > > > >> > > > > > > > > > > > >> > > > On Tue, Jul 24, 2018 at 8:16 AM, Lucas Wang < > > > > > > > > lucasatu...@gmail.com> > > > > > > > > >> > > wrote: > > > > > > > > >> > > > > > > > > > > > >> > > > > Sure, I can summarize the usage of correlation id. > > But > > > > > > before > > > > > > > I > > > > > > > > do > > > > > > > > >> > > that, > > > > > > > > >> > > > it > > > > > > > > >> > > > > seems > > > > > > > > >> > > > > the same out-of-order processing can also happen > to > > > > > Produce > > > > > > > > >> requests > > > > > > > > >> > > sent > > > > > > > > >> > > > > by producers, > > > > > > > > >> > > > > following the same example you described earlier. > > > > > > > > >> > > > > If that's the case, I think this probably > deserves a > > > > > > separate > > > > > > > > doc > > > > > > > > >> and > > > > > > > > >> > > > > design independent of this KIP. > > > > > > > > >> > > > > > > > > > > > > >> > > > > Lucas > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > On Mon, Jul 23, 2018 at 12:39 PM, Dong Lin < > > > > > > > lindon...@gmail.com > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > >> > > > > > > > > > > > > >> > > > > > Hey Lucas, > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Could you update the KIP if you are confident > with > > > the > > > > > > > > approach > > > > > > > > >> > which > > > > > > > > >> > > > > uses > > > > > > > > >> > > > > > correlation id? The idea around correlation id > is > > > kind > > > > > of > > > > > > > > >> scattered > > > > > > > > >> > > > > across > > > > > > > > >> > > > > > multiple emails. It will be useful if other > > reviews > > > > can > > > > > > read > > > > > > > > the > > > > > > > > >> > KIP > > > > > > > > >> > > to > > > > > > > > >> > > > > > understand the latest proposal. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > > > >> > > > > > Dong > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > On Mon, Jul 23, 2018 at 12:32 PM, Mayuresh > Gharat > > < > > > > > > > > >> > > > > > gharatmayures...@gmail.com> wrote: > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > I like the idea of the dequeue implementation > by > > > > > Lucas. > > > > > > > This > > > > > > > > >> will > > > > > > > > >> > > > help > > > > > > > > >> > > > > us > > > > > > > > >> > > > > > > avoid additional queue for controller and > > > additional > > > > > > > configs > > > > > > > > >> in > > > > > > > > >> > > > Kafka. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Thanks, > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Mayuresh > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > On Sun, Jul 22, 2018 at 2:58 AM Becket Qin < > > > > > > > > >> becket....@gmail.com > > > > > > > > >> > > > > > > > > > > >> > > > > wrote: > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > Hi Jun, > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > The usage of correlation ID might still be > > > useful > > > > to > > > > > > > > address > > > > > > > > >> > the > > > > > > > > >> > > > > cases > > > > > > > > >> > > > > > > > that the controller epoch and leader epoch > > check > > > > are > > > > > > not > > > > > > > > >> > > sufficient > > > > > > > > >> > > > > to > > > > > > > > >> > > > > > > > guarantee correct behavior. For example, if > > the > > > > > > > controller > > > > > > > > >> > sends > > > > > > > > >> > > a > > > > > > > > >> > > > > > > > LeaderAndIsrRequest followed by a > > > > > StopReplicaRequest, > > > > > > > and > > > > > > > > >> the > > > > > > > > >> > > > broker > > > > > > > > >> > > > > > > > processes it in the reverse order, the > replica > > > may > > > > > > still > > > > > > > > be > > > > > > > > >> > > wrongly > > > > > > > > >> > > > > > > > recreated, right? > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > Thanks, > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > Jiangjie (Becket) Qin > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > On Jul 22, 2018, at 11:47 AM, Jun Rao < > > > > > > > j...@confluent.io > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > Hmm, since we already use controller epoch > > and > > > > > > leader > > > > > > > > >> epoch > > > > > > > > >> > for > > > > > > > > >> > > > > > > properly > > > > > > > > >> > > > > > > > > caching the latest partition state, do we > > > really > > > > > > need > > > > > > > > >> > > correlation > > > > > > > > >> > > > > id > > > > > > > > >> > > > > > > for > > > > > > > > >> > > > > > > > > ordering the controller requests? > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > Thanks, > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > Jun > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > On Fri, Jul 20, 2018 at 2:18 PM, Becket > Qin > > < > > > > > > > > >> > > > becket....@gmail.com> > > > > > > > > >> > > > > > > > wrote: > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> Lucas and Mayuresh, > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> Good idea. The correlation id should > work. > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> In the ControllerChannelManager, a > request > > > will > > > > > be > > > > > > > > resent > > > > > > > > >> > > until > > > > > > > > >> > > > a > > > > > > > > >> > > > > > > > response > > > > > > > > >> > > > > > > > >> is received. So if the controller to > broker > > > > > > > connection > > > > > > > > >> > > > disconnects > > > > > > > > >> > > > > > > after > > > > > > > > >> > > > > > > > >> controller sends R1_a, but before the > > > response > > > > of > > > > > > > R1_a > > > > > > > > is > > > > > > > > >> > > > > received, > > > > > > > > >> > > > > > a > > > > > > > > >> > > > > > > > >> disconnection may cause the controller to > > > > resend > > > > > > > R1_b. > > > > > > > > >> i.e. > > > > > > > > >> > > > until > > > > > > > > >> > > > > R1 > > > > > > > > >> > > > > > > is > > > > > > > > >> > > > > > > > >> acked, R2 won't be sent by the > controller. > > > > > > > > >> > > > > > > > >> This gives two guarantees: > > > > > > > > >> > > > > > > > >> 1. Correlation id wise: R1_a < R1_b < R2. > > > > > > > > >> > > > > > > > >> 2. On the broker side, when R2 is seen, > R1 > > > must > > > > > > have > > > > > > > > been > > > > > > > > >> > > > > processed > > > > > > > > >> > > > > > at > > > > > > > > >> > > > > > > > >> least once. > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> So on the broker side, with a single > thread > > > > > > > controller > > > > > > > > >> > request > > > > > > > > >> > > > > > > handler, > > > > > > > > >> > > > > > > > the > > > > > > > > >> > > > > > > > >> logic should be: > > > > > > > > >> > > > > > > > >> 1. Process what ever request seen in the > > > > > controller > > > > > > > > >> request > > > > > > > > >> > > > queue > > > > > > > > >> > > > > > > > >> 2. For the given epoch, drop request if > its > > > > > > > correlation > > > > > > > > >> id > > > > > > > > >> > is > > > > > > > > >> > > > > > smaller > > > > > > > > >> > > > > > > > than > > > > > > > > >> > > > > > > > >> that of the last processed request. > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> Thanks, > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> Jiangjie (Becket) Qin > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> On Fri, Jul 20, 2018 at 8:07 AM, Jun Rao > < > > > > > > > > >> j...@confluent.io> > > > > > > > > >> > > > > wrote: > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >>> I agree that there is no strong ordering > > > when > > > > > > there > > > > > > > > are > > > > > > > > >> > more > > > > > > > > >> > > > than > > > > > > > > >> > > > > > one > > > > > > > > >> > > > > > > > >>> socket connections. Currently, we rely > on > > > > > > > > >> controllerEpoch > > > > > > > > >> > and > > > > > > > > >> > > > > > > > leaderEpoch > > > > > > > > >> > > > > > > > >>> to ensure that the receiving broker > picks > > up > > > > the > > > > > > > > latest > > > > > > > > >> > state > > > > > > > > >> > > > for > > > > > > > > >> > > > > > > each > > > > > > > > >> > > > > > > > >>> partition. > > > > > > > > >> > > > > > > > >>> > > > > > > > > >> > > > > > > > >>> One potential issue with the dequeue > > > approach > > > > is > > > > > > > that > > > > > > > > if > > > > > > > > >> > the > > > > > > > > >> > > > > queue > > > > > > > > >> > > > > > is > > > > > > > > >> > > > > > > > >> full, > > > > > > > > >> > > > > > > > >>> there is no guarantee that the > controller > > > > > requests > > > > > > > > will > > > > > > > > >> be > > > > > > > > >> > > > > enqueued > > > > > > > > >> > > > > > > > >>> quickly. > > > > > > > > >> > > > > > > > >>> > > > > > > > > >> > > > > > > > >>> Thanks, > > > > > > > > >> > > > > > > > >>> > > > > > > > > >> > > > > > > > >>> Jun > > > > > > > > >> > > > > > > > >>> > > > > > > > > >> > > > > > > > >>> On Fri, Jul 20, 2018 at 5:25 AM, > Mayuresh > > > > > Gharat < > > > > > > > > >> > > > > > > > >>> gharatmayures...@gmail.com > > > > > > > > >> > > > > > > > >>>> wrote: > > > > > > > > >> > > > > > > > >>> > > > > > > > > >> > > > > > > > >>>> Yea, the correlationId is only set to 0 > > in > > > > the > > > > > > > > >> > NetworkClient > > > > > > > > >> > > > > > > > >> constructor. > > > > > > > > >> > > > > > > > >>>> Since we reuse the same NetworkClient > > > between > > > > > > > > >> Controller > > > > > > > > >> > and > > > > > > > > >> > > > the > > > > > > > > >> > > > > > > > >> broker, > > > > > > > > >> > > > > > > > >>> a > > > > > > > > >> > > > > > > > >>>> disconnection should not cause it to > > reset > > > to > > > > > 0, > > > > > > in > > > > > > > > >> which > > > > > > > > >> > > case > > > > > > > > >> > > > > it > > > > > > > > >> > > > > > > can > > > > > > > > >> > > > > > > > >> be > > > > > > > > >> > > > > > > > >>>> used to reject obsolete requests. > > > > > > > > >> > > > > > > > >>>> > > > > > > > > >> > > > > > > > >>>> Thanks, > > > > > > > > >> > > > > > > > >>>> > > > > > > > > >> > > > > > > > >>>> Mayuresh > > > > > > > > >> > > > > > > > >>>> > > > > > > > > >> > > > > > > > >>>> On Thu, Jul 19, 2018 at 1:52 PM Lucas > > Wang > > > < > > > > > > > > >> > > > > lucasatu...@gmail.com > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > >>> wrote: > > > > > > > > >> > > > > > > > >>>> > > > > > > > > >> > > > > > > > >>>>> @Dong, > > > > > > > > >> > > > > > > > >>>>> Great example and explanation, thanks! > > > > > > > > >> > > > > > > > >>>>> > > > > > > > > >> > > > > > > > >>>>> @All > > > > > > > > >> > > > > > > > >>>>> Regarding the example given by Dong, > it > > > > seems > > > > > > even > > > > > > > > if > > > > > > > > >> we > > > > > > > > >> > > use > > > > > > > > >> > > > a > > > > > > > > >> > > > > > > queue, > > > > > > > > >> > > > > > > > >>>> and a > > > > > > > > >> > > > > > > > >>>>> dedicated controller request handling > > > > thread, > > > > > > > > >> > > > > > > > >>>>> the same result can still happen > because > > > > R1_a > > > > > > will > > > > > > > > be > > > > > > > > >> > sent > > > > > > > > >> > > on > > > > > > > > >> > > > > one > > > > > > > > >> > > > > > > > >>>>> connection, and R1_b & R2 will be sent > > on > > > a > > > > > > > > different > > > > > > > > >> > > > > connection, > > > > > > > > >> > > > > > > > >>>>> and there is no ordering between > > different > > > > > > > > >> connections on > > > > > > > > >> > > the > > > > > > > > >> > > > > > > broker > > > > > > > > >> > > > > > > > >>>> side. > > > > > > > > >> > > > > > > > >>>>> I was discussing with Mayuresh > offline, > > > and > > > > it > > > > > > > seems > > > > > > > > >> > > > > correlation > > > > > > > > >> > > > > > id > > > > > > > > >> > > > > > > > >>>> within > > > > > > > > >> > > > > > > > >>>>> the same NetworkClient object is > > > > monotonically > > > > > > > > >> increasing > > > > > > > > >> > > and > > > > > > > > >> > > > > > never > > > > > > > > >> > > > > > > > >>>> reset, > > > > > > > > >> > > > > > > > >>>>> hence a broker can leverage that to > > > properly > > > > > > > reject > > > > > > > > >> > > obsolete > > > > > > > > >> > > > > > > > >> requests. > > > > > > > > >> > > > > > > > >>>>> Thoughts? > > > > > > > > >> > > > > > > > >>>>> > > > > > > > > >> > > > > > > > >>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>> Lucas > > > > > > > > >> > > > > > > > >>>>> > > > > > > > > >> > > > > > > > >>>>> On Thu, Jul 19, 2018 at 12:11 PM, > > Mayuresh > > > > > > Gharat > > > > > > > < > > > > > > > > >> > > > > > > > >>>>> gharatmayures...@gmail.com> wrote: > > > > > > > > >> > > > > > > > >>>>> > > > > > > > > >> > > > > > > > >>>>>> Actually nvm, correlationId is reset > in > > > > case > > > > > of > > > > > > > > >> > connection > > > > > > > > >> > > > > > loss, I > > > > > > > > >> > > > > > > > >>>> think. > > > > > > > > >> > > > > > > > >>>>>> > > > > > > > > >> > > > > > > > >>>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>>> > > > > > > > > >> > > > > > > > >>>>>> Mayuresh > > > > > > > > >> > > > > > > > >>>>>> > > > > > > > > >> > > > > > > > >>>>>> On Thu, Jul 19, 2018 at 11:11 AM > > Mayuresh > > > > > > Gharat > > > > > > > < > > > > > > > > >> > > > > > > > >>>>>> gharatmayures...@gmail.com> > > > > > > > > >> > > > > > > > >>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>> > > > > > > > > >> > > > > > > > >>>>>>> I agree with Dong that out-of-order > > > > > processing > > > > > > > can > > > > > > > > >> > happen > > > > > > > > >> > > > > with > > > > > > > > >> > > > > > > > >>>> having 2 > > > > > > > > >> > > > > > > > >>>>>>> separate queues as well and it can > > even > > > > > happen > > > > > > > > >> today. > > > > > > > > >> > > > > > > > >>>>>>> Can we use the correlationId in the > > > > request > > > > > > from > > > > > > > > the > > > > > > > > >> > > > > controller > > > > > > > > >> > > > > > > > >> to > > > > > > > > >> > > > > > > > >>>> the > > > > > > > > >> > > > > > > > >>>>>>> broker to handle ordering ? > > > > > > > > >> > > > > > > > >>>>>>> > > > > > > > > >> > > > > > > > >>>>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>>>> > > > > > > > > >> > > > > > > > >>>>>>> Mayuresh > > > > > > > > >> > > > > > > > >>>>>>> > > > > > > > > >> > > > > > > > >>>>>>> > > > > > > > > >> > > > > > > > >>>>>>> On Thu, Jul 19, 2018 at 6:41 AM > Becket > > > > Qin < > > > > > > > > >> > > > > > becket....@gmail.com > > > > > > > > >> > > > > > > > >>> > > > > > > > > >> > > > > > > > >>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>> Good point, Joel. I agree that a > > > > dedicated > > > > > > > > >> controller > > > > > > > > >> > > > > request > > > > > > > > >> > > > > > > > >>>> handling > > > > > > > > >> > > > > > > > >>>>>>>> thread would be a better isolation. > > It > > > > also > > > > > > > > solves > > > > > > > > >> the > > > > > > > > >> > > > > > > > >> reordering > > > > > > > > >> > > > > > > > >>>>> issue. > > > > > > > > >> > > > > > > > >>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>> On Thu, Jul 19, 2018 at 2:23 PM, > Joel > > > > > Koshy < > > > > > > > > >> > > > > > > > >> jjkosh...@gmail.com> > > > > > > > > >> > > > > > > > >>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>> Good example. I think this > scenario > > > can > > > > > > occur > > > > > > > in > > > > > > > > >> the > > > > > > > > >> > > > > current > > > > > > > > >> > > > > > > > >>> code > > > > > > > > >> > > > > > > > >>>> as > > > > > > > > >> > > > > > > > >>>>>>>> well > > > > > > > > >> > > > > > > > >>>>>>>>> but with even lower probability > > given > > > > that > > > > > > > there > > > > > > > > >> are > > > > > > > > >> > > > other > > > > > > > > >> > > > > > > > >>>>>>>> non-controller > > > > > > > > >> > > > > > > > >>>>>>>>> requests interleaved. It is still > > > > sketchy > > > > > > > though > > > > > > > > >> and > > > > > > > > >> > I > > > > > > > > >> > > > > think > > > > > > > > >> > > > > > a > > > > > > > > >> > > > > > > > >>>> safer > > > > > > > > >> > > > > > > > >>>>>>>>> approach would be separate queues > > and > > > > > > pinning > > > > > > > > >> > > controller > > > > > > > > >> > > > > > > > >> request > > > > > > > > >> > > > > > > > >>>>>>>> handling > > > > > > > > >> > > > > > > > >>>>>>>>> to one handler thread. > > > > > > > > >> > > > > > > > >>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>> On Wed, Jul 18, 2018 at 11:12 PM, > > Dong > > > > > Lin < > > > > > > > > >> > > > > > > > >> lindon...@gmail.com > > > > > > > > >> > > > > > > > >>>> > > > > > > > > >> > > > > > > > >>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>> Hey Becket, > > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>> I think you are right that there > > may > > > be > > > > > > > > >> out-of-order > > > > > > > > >> > > > > > > > >>> processing. > > > > > > > > >> > > > > > > > >>>>>>>> However, > > > > > > > > >> > > > > > > > >>>>>>>>>> it seems that out-of-order > > processing > > > > may > > > > > > > also > > > > > > > > >> > happen > > > > > > > > >> > > > even > > > > > > > > >> > > > > > > > >> if > > > > > > > > >> > > > > > > > >>> we > > > > > > > > >> > > > > > > > >>>>>> use a > > > > > > > > >> > > > > > > > >>>>>>>>>> separate queue. > > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>> Here is the example: > > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>> - Controller sends R1 and got > > > > > disconnected > > > > > > > > before > > > > > > > > >> > > > > receiving > > > > > > > > >> > > > > > > > >>>>>> response. > > > > > > > > >> > > > > > > > >>>>>>>>> Then > > > > > > > > >> > > > > > > > >>>>>>>>>> it reconnects and sends R2. Both > > > > requests > > > > > > now > > > > > > > > >> stay > > > > > > > > >> > in > > > > > > > > >> > > > the > > > > > > > > >> > > > > > > > >>>>> controller > > > > > > > > >> > > > > > > > >>>>>>>>>> request queue in the order they > are > > > > sent. > > > > > > > > >> > > > > > > > >>>>>>>>>> - thread1 takes R1_a from the > > request > > > > > queue > > > > > > > and > > > > > > > > >> then > > > > > > > > >> > > > > thread2 > > > > > > > > >> > > > > > > > >>>> takes > > > > > > > > >> > > > > > > > >>>>>> R2 > > > > > > > > >> > > > > > > > >>>>>>>>> from > > > > > > > > >> > > > > > > > >>>>>>>>>> the request queue almost at the > > same > > > > > time. > > > > > > > > >> > > > > > > > >>>>>>>>>> - So R1_a and R2 are processed in > > > > > parallel. > > > > > > > > >> There is > > > > > > > > >> > > > > chance > > > > > > > > >> > > > > > > > >>> that > > > > > > > > >> > > > > > > > >>>>>> R2's > > > > > > > > >> > > > > > > > >>>>>>>>>> processing is completed before > R1. > > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>> If out-of-order processing can > > happen > > > > for > > > > > > > both > > > > > > > > >> > > > approaches > > > > > > > > >> > > > > > > > >> with > > > > > > > > >> > > > > > > > >>>>> very > > > > > > > > >> > > > > > > > >>>>>>>> low > > > > > > > > >> > > > > > > > >>>>>>>>>> probability, it may not be > > worthwhile > > > > to > > > > > > add > > > > > > > > the > > > > > > > > >> > extra > > > > > > > > >> > > > > > > > >> queue. > > > > > > > > >> > > > > > > > >>>> What > > > > > > > > >> > > > > > > > >>>>>> do > > > > > > > > >> > > > > > > > >>>>>>>> you > > > > > > > > >> > > > > > > > >>>>>>>>>> think? > > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>>>>>>> Dong > > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>> On Wed, Jul 18, 2018 at 6:17 PM, > > > Becket > > > > > > Qin < > > > > > > > > >> > > > > > > > >>>> becket....@gmail.com > > > > > > > > >> > > > > > > > >>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>> Hi Mayuresh/Joel, > > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>> Using the request channel as a > > > dequeue > > > > > was > > > > > > > > >> bright > > > > > > > > >> > up > > > > > > > > >> > > > some > > > > > > > > >> > > > > > > > >>> time > > > > > > > > >> > > > > > > > >>>>> ago > > > > > > > > >> > > > > > > > >>>>>>>> when > > > > > > > > >> > > > > > > > >>>>>>>>>> we > > > > > > > > >> > > > > > > > >>>>>>>>>>> initially thinking of > prioritizing > > > the > > > > > > > > request. > > > > > > > > >> The > > > > > > > > >> > > > > > > > >> concern > > > > > > > > >> > > > > > > > >>>> was > > > > > > > > >> > > > > > > > >>>>>> that > > > > > > > > >> > > > > > > > >>>>>>>>> the > > > > > > > > >> > > > > > > > >>>>>>>>>>> controller requests are supposed > > to > > > be > > > > > > > > >> processed in > > > > > > > > >> > > > > order. > > > > > > > > >> > > > > > > > >>> If > > > > > > > > >> > > > > > > > >>>> we > > > > > > > > >> > > > > > > > >>>>>> can > > > > > > > > >> > > > > > > > >>>>>>>>>> ensure > > > > > > > > >> > > > > > > > >>>>>>>>>>> that there is one controller > > request > > > > in > > > > > > the > > > > > > > > >> request > > > > > > > > >> > > > > > > > >> channel, > > > > > > > > >> > > > > > > > >>>> the > > > > > > > > >> > > > > > > > >>>>>>>> order > > > > > > > > >> > > > > > > > >>>>>>>>> is > > > > > > > > >> > > > > > > > >>>>>>>>>>> not a concern. But in cases that > > > there > > > > > are > > > > > > > > more > > > > > > > > >> > than > > > > > > > > >> > > > one > > > > > > > > >> > > > > > > > >>>>>> controller > > > > > > > > >> > > > > > > > >>>>>>>>>> request > > > > > > > > >> > > > > > > > >>>>>>>>>>> inserted into the queue, the > > > > controller > > > > > > > > request > > > > > > > > >> > order > > > > > > > > >> > > > may > > > > > > > > >> > > > > > > > >>>> change > > > > > > > > >> > > > > > > > >>>>>> and > > > > > > > > >> > > > > > > > >>>>>>>>>> cause > > > > > > > > >> > > > > > > > >>>>>>>>>>> problem. For example, think > about > > > the > > > > > > > > following > > > > > > > > >> > > > sequence: > > > > > > > > >> > > > > > > > >>>>>>>>>>> 1. Controller successfully sent > a > > > > > request > > > > > > R1 > > > > > > > > to > > > > > > > > >> > > broker > > > > > > > > >> > > > > > > > >>>>>>>>>>> 2. Broker receives R1 and put > the > > > > > request > > > > > > to > > > > > > > > the > > > > > > > > >> > head > > > > > > > > >> > > > of > > > > > > > > >> > > > > > > > >> the > > > > > > > > >> > > > > > > > >>>>>> request > > > > > > > > >> > > > > > > > >>>>>>>>>> queue. > > > > > > > > >> > > > > > > > >>>>>>>>>>> 3. Controller to broker > connection > > > > > failed > > > > > > > and > > > > > > > > >> the > > > > > > > > >> > > > > > > > >> controller > > > > > > > > >> > > > > > > > >>>>>>>>> reconnected > > > > > > > > >> > > > > > > > >>>>>>>>>> to > > > > > > > > >> > > > > > > > >>>>>>>>>>> the broker. > > > > > > > > >> > > > > > > > >>>>>>>>>>> 4. Controller sends a request R2 > > to > > > > the > > > > > > > broker > > > > > > > > >> > > > > > > > >>>>>>>>>>> 5. Broker receives R2 and add it > > to > > > > the > > > > > > head > > > > > > > > of > > > > > > > > >> the > > > > > > > > >> > > > > > > > >> request > > > > > > > > >> > > > > > > > >>>>> queue. > > > > > > > > >> > > > > > > > >>>>>>>>>>> Now on the broker side, R2 will > be > > > > > > processed > > > > > > > > >> before > > > > > > > > >> > > R1 > > > > > > > > >> > > > is > > > > > > > > >> > > > > > > > >>>>>> processed, > > > > > > > > >> > > > > > > > >>>>>>>>>> which > > > > > > > > >> > > > > > > > >>>>>>>>>>> may cause problem. > > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>> Jiangjie (Becket) Qin > > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>> On Thu, Jul 19, 2018 at 3:23 AM, > > > Joel > > > > > > Koshy > > > > > > > < > > > > > > > > >> > > > > > > > >>>>> jjkosh...@gmail.com> > > > > > > > > >> > > > > > > > >>>>>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>> @Mayuresh - I like your idea. > It > > > > > appears > > > > > > to > > > > > > > > be > > > > > > > > >> a > > > > > > > > >> > > > simpler > > > > > > > > >> > > > > > > > >>>> less > > > > > > > > >> > > > > > > > >>>>>>>>> invasive > > > > > > > > >> > > > > > > > >>>>>>>>>>>> alternative and it should work. > > > > > > > > >> Jun/Becket/others, > > > > > > > > >> > > do > > > > > > > > >> > > > > > > > >> you > > > > > > > > >> > > > > > > > >>>> see > > > > > > > > >> > > > > > > > >>>>>> any > > > > > > > > >> > > > > > > > >>>>>>>>>>> pitfalls > > > > > > > > >> > > > > > > > >>>>>>>>>>>> with this approach? > > > > > > > > >> > > > > > > > >>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>> On Wed, Jul 18, 2018 at 12:03 > PM, > > > > Lucas > > > > > > > Wang > > > > > > > > < > > > > > > > > >> > > > > > > > >>>>>>>> lucasatu...@gmail.com> > > > > > > > > >> > > > > > > > >>>>>>>>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> @Mayuresh, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> That's a very interesting idea > > > that > > > > I > > > > > > > > haven't > > > > > > > > >> > > thought > > > > > > > > >> > > > > > > > >>>>> before. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> It seems to solve our problem > at > > > > hand > > > > > > > pretty > > > > > > > > >> > well, > > > > > > > > >> > > > and > > > > > > > > >> > > > > > > > >>>> also > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> avoids the need to have a new > > size > > > > > > metric > > > > > > > > and > > > > > > > > >> > > > capacity > > > > > > > > >> > > > > > > > >>>>> config > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> for the controller request > > queue. > > > In > > > > > > fact, > > > > > > > > if > > > > > > > > >> we > > > > > > > > >> > > were > > > > > > > > >> > > > > > > > >> to > > > > > > > > >> > > > > > > > >>>>> adopt > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> this design, there is no > public > > > > > > interface > > > > > > > > >> change, > > > > > > > > >> > > and > > > > > > > > >> > > > > > > > >> we > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> probably don't need a KIP. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> Also implementation wise, it > > seems > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> the java class > > LinkedBlockingQueue > > > > can > > > > > > > > readily > > > > > > > > >> > > > satisfy > > > > > > > > >> > > > > > > > >>> the > > > > > > > > >> > > > > > > > >>>>>>>>>> requirement > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> by supporting a capacity, and > > also > > > > > > > allowing > > > > > > > > >> > > inserting > > > > > > > > >> > > > > > > > >> at > > > > > > > > >> > > > > > > > >>>>> both > > > > > > > > >> > > > > > > > >>>>>>>> ends. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> My only concern is that this > > > design > > > > is > > > > > > > tied > > > > > > > > to > > > > > > > > >> > the > > > > > > > > >> > > > > > > > >>>>> coincidence > > > > > > > > >> > > > > > > > >>>>>>>> that > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> we have two request priorities > > and > > > > > there > > > > > > > are > > > > > > > > >> two > > > > > > > > >> > > ends > > > > > > > > >> > > > > > > > >>> to a > > > > > > > > >> > > > > > > > >>>>>>>> deque. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> Hence by using the proposed > > > design, > > > > it > > > > > > > seems > > > > > > > > >> the > > > > > > > > >> > > > > > > > >> network > > > > > > > > >> > > > > > > > >>>>> layer > > > > > > > > >> > > > > > > > >>>>>>>> is > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> more tightly coupled with > upper > > > > layer > > > > > > > logic, > > > > > > > > >> e.g. > > > > > > > > >> > > if > > > > > > > > >> > > > > > > > >> we > > > > > > > > >> > > > > > > > >>>> were > > > > > > > > >> > > > > > > > >>>>>> to > > > > > > > > >> > > > > > > > >>>>>>>> add > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> an extra priority level in the > > > > future > > > > > > for > > > > > > > > some > > > > > > > > >> > > > reason, > > > > > > > > >> > > > > > > > >>> we > > > > > > > > >> > > > > > > > >>>>>> would > > > > > > > > >> > > > > > > > >>>>>>>>>>> probably > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> need to go back to the design > of > > > > > > separate > > > > > > > > >> queues, > > > > > > > > >> > > one > > > > > > > > >> > > > > > > > >>> for > > > > > > > > >> > > > > > > > >>>>> each > > > > > > > > >> > > > > > > > >>>>>>>>>> priority > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> level. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> In summary, I'm ok with both > > > designs > > > > > and > > > > > > > > lean > > > > > > > > >> > > toward > > > > > > > > >> > > > > > > > >>> your > > > > > > > > >> > > > > > > > >>>>>>>> suggested > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> approach. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> Let's hear what others think. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> @Becket, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> In light of Mayuresh's > suggested > > > new > > > > > > > design, > > > > > > > > >> I'm > > > > > > > > >> > > > > > > > >>> answering > > > > > > > > >> > > > > > > > >>>>>> your > > > > > > > > >> > > > > > > > >>>>>>>>>>> question > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> only in the context > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> of the current KIP design: I > > think > > > > > your > > > > > > > > >> > suggestion > > > > > > > > >> > > > > > > > >> makes > > > > > > > > >> > > > > > > > >>>>>> sense, > > > > > > > > >> > > > > > > > >>>>>>>> and > > > > > > > > >> > > > > > > > >>>>>>>>>> I'm > > > > > > > > >> > > > > > > > >>>>>>>>>>>> ok > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> with removing the capacity > > config > > > > and > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> just relying on the default > > value > > > of > > > > > 20 > > > > > > > > being > > > > > > > > >> > > > > > > > >> sufficient > > > > > > > > >> > > > > > > > >>>>>> enough. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> Lucas > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> On Wed, Jul 18, 2018 at 9:57 > AM, > > > > > > Mayuresh > > > > > > > > >> Gharat > > > > > > > > >> > < > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> gharatmayures...@gmail.com > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> Hi Lucas, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> Seems like the main intent > here > > > is > > > > to > > > > > > > > >> prioritize > > > > > > > > >> > > the > > > > > > > > >> > > > > > > > >>>>>>>> controller > > > > > > > > >> > > > > > > > >>>>>>>>>>> request > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> over any other requests. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> In that case, we can change > the > > > > > request > > > > > > > > queue > > > > > > > > >> > to a > > > > > > > > >> > > > > > > > >>>>> dequeue, > > > > > > > > >> > > > > > > > >>>>>>>> where > > > > > > > > >> > > > > > > > >>>>>>>>>> you > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> always insert the normal > > requests > > > > > > > (produce, > > > > > > > > >> > > > > > > > >>>> consume,..etc) > > > > > > > > >> > > > > > > > >>>>>> to > > > > > > > > >> > > > > > > > >>>>>>>> the > > > > > > > > >> > > > > > > > >>>>>>>>>> end > > > > > > > > >> > > > > > > > >>>>>>>>>>>> of > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> the dequeue, but if its a > > > > controller > > > > > > > > request, > > > > > > > > >> > you > > > > > > > > >> > > > > > > > >>> insert > > > > > > > > >> > > > > > > > >>>>> it > > > > > > > > >> > > > > > > > >>>>>> to > > > > > > > > >> > > > > > > > >>>>>>>>> the > > > > > > > > >> > > > > > > > >>>>>>>>>>> head > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> of > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> the queue. This ensures that > > the > > > > > > > controller > > > > > > > > >> > > request > > > > > > > > >> > > > > > > > >>> will > > > > > > > > >> > > > > > > > >>>>> be > > > > > > > > >> > > > > > > > >>>>>>>> given > > > > > > > > >> > > > > > > > >>>>>>>>>>>> higher > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> priority over other requests. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> Also since we only read one > > > request > > > > > > from > > > > > > > > the > > > > > > > > >> > > socket > > > > > > > > >> > > > > > > > >>> and > > > > > > > > >> > > > > > > > >>>>> mute > > > > > > > > >> > > > > > > > >>>>>>>> it > > > > > > > > >> > > > > > > > >>>>>>>>> and > > > > > > > > >> > > > > > > > >>>>>>>>>>>> only > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> unmute it after handling the > > > > request, > > > > > > > this > > > > > > > > >> would > > > > > > > > >> > > > > > > > >>> ensure > > > > > > > > >> > > > > > > > >>>>> that > > > > > > > > >> > > > > > > > >>>>>>>> we > > > > > > > > >> > > > > > > > >>>>>>>>>> don't > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> handle controller requests > out > > of > > > > > > order. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> With this approach we can > avoid > > > the > > > > > > > second > > > > > > > > >> queue > > > > > > > > >> > > and > > > > > > > > >> > > > > > > > >>> the > > > > > > > > >> > > > > > > > >>>>>>>>> additional > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> config > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> for the size of the queue. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> What do you think ? > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> Mayuresh > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> On Wed, Jul 18, 2018 at 3:05 > AM > > > > > Becket > > > > > > > Qin > > > > > > > > < > > > > > > > > >> > > > > > > > >>>>>>>> becket....@gmail.com > > > > > > > > >> > > > > > > > >>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> Hey Joel, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> Thank for the detail > > > explanation. > > > > I > > > > > > > agree > > > > > > > > >> the > > > > > > > > >> > > > > > > > >>> current > > > > > > > > >> > > > > > > > >>>>>> design > > > > > > > > >> > > > > > > > >>>>>>>>>> makes > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> sense. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> My confusion is about > whether > > > the > > > > > new > > > > > > > > config > > > > > > > > >> > for > > > > > > > > >> > > > > > > > >> the > > > > > > > > >> > > > > > > > >>>>>>>> controller > > > > > > > > >> > > > > > > > >>>>>>>>>>> queue > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> capacity is necessary. I > > cannot > > > > > think > > > > > > > of a > > > > > > > > >> case > > > > > > > > >> > > in > > > > > > > > >> > > > > > > > >>>> which > > > > > > > > >> > > > > > > > >>>>>>>> users > > > > > > > > >> > > > > > > > >>>>>>>>>>> would > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> change > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> it. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> Jiangjie (Becket) Qin > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> On Wed, Jul 18, 2018 at 6:00 > > PM, > > > > > > Becket > > > > > > > > Qin > > > > > > > > >> < > > > > > > > > >> > > > > > > > >>>>>>>>>> becket....@gmail.com> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> Hi Lucas, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> I guess my question can be > > > > > rephrased > > > > > > to > > > > > > > > >> "do we > > > > > > > > >> > > > > > > > >>>> expect > > > > > > > > >> > > > > > > > >>>>>>>> user to > > > > > > > > >> > > > > > > > >>>>>>>>>>> ever > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> change > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> the controller request > queue > > > > > > capacity"? > > > > > > > > If > > > > > > > > >> we > > > > > > > > >> > > > > > > > >>> agree > > > > > > > > >> > > > > > > > >>>>> that > > > > > > > > >> > > > > > > > >>>>>>>> 20 > > > > > > > > >> > > > > > > > >>>>>>>>> is > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> already > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> a > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> very generous default > number > > > and > > > > we > > > > > > do > > > > > > > > not > > > > > > > > >> > > > > > > > >> expect > > > > > > > > >> > > > > > > > >>>> user > > > > > > > > >> > > > > > > > >>>>>> to > > > > > > > > >> > > > > > > > >>>>>>>>>> change > > > > > > > > >> > > > > > > > >>>>>>>>>>>> it, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> is > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> it > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> still necessary to expose > > this > > > > as a > > > > > > > > config? > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> Jiangjie (Becket) Qin > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> On Wed, Jul 18, 2018 at > 2:29 > > > AM, > > > > > > Lucas > > > > > > > > >> Wang < > > > > > > > > >> > > > > > > > >>>>>>>>>>> lucasatu...@gmail.com > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> @Becket > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> 1. Thanks for the comment. > > You > > > > are > > > > > > > right > > > > > > > > >> that > > > > > > > > >> > > > > > > > >>>>> normally > > > > > > > > >> > > > > > > > >>>>>>>> there > > > > > > > > >> > > > > > > > >>>>>>>>>>>> should > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> be > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> just > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> one controller request > > because > > > > of > > > > > > > > muting, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> and I had NOT intended to > > say > > > > > there > > > > > > > > would > > > > > > > > >> be > > > > > > > > >> > > > > > > > >> many > > > > > > > > >> > > > > > > > >>>>>>>> enqueued > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> controller > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> requests. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> I went through the KIP > > again, > > > > and > > > > > > I'm > > > > > > > > not > > > > > > > > >> > sure > > > > > > > > >> > > > > > > > >>>> which > > > > > > > > >> > > > > > > > >>>>>> part > > > > > > > > >> > > > > > > > >>>>>>>>>>> conveys > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> that > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> info. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> I'd be happy to revise if > > you > > > > > point > > > > > > it > > > > > > > > out > > > > > > > > >> > the > > > > > > > > >> > > > > > > > >>>>> section. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> 2. Though it should not > > happen > > > > in > > > > > > > normal > > > > > > > > >> > > > > > > > >>>> conditions, > > > > > > > > >> > > > > > > > >>>>>> the > > > > > > > > >> > > > > > > > >>>>>>>>>> current > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> design > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> does not preclude multiple > > > > > > controllers > > > > > > > > >> > running > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> at the same time, hence if > > we > > > > > don't > > > > > > > have > > > > > > > > >> the > > > > > > > > >> > > > > > > > >>>>> controller > > > > > > > > >> > > > > > > > >>>>>>>>> queue > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> capacity > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> config and simply make its > > > > > capacity > > > > > > to > > > > > > > > be > > > > > > > > >> 1, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> network threads handling > > > > requests > > > > > > from > > > > > > > > >> > > > > > > > >> different > > > > > > > > >> > > > > > > > >>>>>>>> controllers > > > > > > > > >> > > > > > > > >>>>>>>>>>> will > > > > > > > > >> > > > > > > > >>>>>>>>>>>> be > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> blocked during those > > > troublesome > > > > > > > times, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> which is probably not what > > we > > > > > want. > > > > > > On > > > > > > > > the > > > > > > > > >> > > > > > > > >> other > > > > > > > > >> > > > > > > > >>>>> hand, > > > > > > > > >> > > > > > > > >>>>>>>>> adding > > > > > > > > >> > > > > > > > >>>>>>>>>>> the > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> extra > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> config with a default > value, > > > say > > > > > 20, > > > > > > > > >> guards > > > > > > > > >> > us > > > > > > > > >> > > > > > > > >>> from > > > > > > > > >> > > > > > > > >>>>>>>> issues > > > > > > > > >> > > > > > > > >>>>>>>>> in > > > > > > > > >> > > > > > > > >>>>>>>>>>>> those > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> troublesome times, and IMO > > > there > > > > > > isn't > > > > > > > > >> much > > > > > > > > >> > > > > > > > >>>> downside > > > > > > > > >> > > > > > > > >>>>> of > > > > > > > > >> > > > > > > > >>>>>>>>> adding > > > > > > > > >> > > > > > > > >>>>>>>>>>> the > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> extra > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> config. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> @Mayuresh > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> Good catch, this sentence > is > > > an > > > > > > > obsolete > > > > > > > > >> > > > > > > > >>> statement > > > > > > > > >> > > > > > > > >>>>>> based > > > > > > > > >> > > > > > > > >>>>>>>> on > > > > > > > > >> > > > > > > > >>>>>>>>> a > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> previous > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> design. I've revised the > > > wording > > > > > in > > > > > > > the > > > > > > > > >> KIP. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> Lucas > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> On Tue, Jul 17, 2018 at > > 10:33 > > > > AM, > > > > > > > > Mayuresh > > > > > > > > >> > > > > > > > >>> Gharat < > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > gharatmayures...@gmail.com> > > > > > wrote: > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> Hi Lucas, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks for the KIP. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> I am trying to understand > > why > > > > you > > > > > > > think > > > > > > > > >> "The > > > > > > > > >> > > > > > > > >>>> memory > > > > > > > > >> > > > > > > > >>>>>>>>>>> consumption > > > > > > > > >> > > > > > > > >>>>>>>>>>>>> can > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> rise > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> given the total number of > > > > queued > > > > > > > > requests > > > > > > > > >> > can > > > > > > > > >> > > > > > > > >>> go > > > > > > > > >> > > > > > > > >>>> up > > > > > > > > >> > > > > > > > >>>>>> to > > > > > > > > >> > > > > > > > >>>>>>>> 2x" > > > > > > > > >> > > > > > > > >>>>>>>>>> in > > > > > > > > >> > > > > > > > >>>>>>>>>>>> the > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> impact > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> section. Normally the > > > requests > > > > > from > > > > > > > > >> > > > > > > > >> controller > > > > > > > > >> > > > > > > > >>>> to a > > > > > > > > >> > > > > > > > >>>>>>>> Broker > > > > > > > > >> > > > > > > > >>>>>>>>>> are > > > > > > > > >> > > > > > > > >>>>>>>>>>>> not > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> high > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> volume, right ? > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks, > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> Mayuresh > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> On Tue, Jul 17, 2018 at > > 5:06 > > > AM > > > > > > > Becket > > > > > > > > >> Qin < > > > > > > > > >> > > > > > > > >>>>>>>>>>>> becket....@gmail.com> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> wrote: > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> Thanks for the KIP, > Lucas. > > > > > > > Separating > > > > > > > > >> the > > > > > > > > >> > > > > > > > >>>> control > > > > > > > > >> > > > > > > > >>>>>>>> plane > > > > > > > > >> > > > > > > > >>>>>>>>>> from > > > > > > > > >> > > > > > > > >>>>>>>>>>>> the > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> data > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> plane > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> makes a lot of sense. > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> In the KIP you mentioned > > > that > > > > > the > > > > > > > > >> > > > > > > > >> controller > > > > > > > > >> > > > > > > > >>>>>> request > > > > > > > > >> > > > > > > > >>>>>>>>> queue > > > > > > > > >> > > > > > > > >>>>>>>>>>> may > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> have > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> many > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> requests in it. Will > this > > > be a > > > > > > > common > > > > > > > > >> case? > > > > > > > > >> > > > > > > > >>> The > > > > > > > > >> > > > > > > > >>>>>>>>> controller > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>> requests > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> still > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> goes through the > > > SocketServer. > > > > > The > > > > > > > > >> > > > > > > > >>> SocketServer > > > > > > > > >> > > > > > > > >>>>>> will > > > > > > > > >> > > > > > > > >>>>>>>>> mute > > > > > > > > >> > > > > > > > >>>>>>>>>>> the > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> channel > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>> once > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> a request is read and > put > > > into > > > > > the > > > > > > > > >> request > > > > > > > > >> > > > > > > > >>>>> channel. > > > > > > > > >> > > > > > > > >>>>>>>> So > > > > > > > > >> > > > > > > > >>>>>>>>>>>> assuming > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>> there > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>> is > > > > > > > > >> > > > > > > > >>>>>>>>>>>>>>>>>>> only one connection > > between > > > > > > > controller > > > > > > > > >> and > > > > > > > > >> > > > > > > > >>> each > > > > > > > > >> > > > > > > > >>>>>>>> broker, > > > > > > > > >> > > > > > > > >>>>>>>>> on > > > > > > > > >> > > > > > > > >>>>>>>>>>> the > > > > > > > > >> > > > > > > > > > > > > > > >
