Hi Eno, Sorry for the delay in getting the experiment results. Here is a link to the positive impact achieved by implementing the proposed change: https://docs.google.com/document/d/1ge2jjp5aPTBber6zaIT9AdhWFWUENJ3JO6Zyu4f9tgQ/edit?usp=sharing Please take a look when you have time and let me know your feedback.
Regards, Lucas On Tue, Jun 26, 2018 at 9:52 AM, Harsha <ka...@harsha.io> wrote: > Thanks for the pointer. Will take a look might suit our requirements > better. > > Thanks, > Harsha > > On Mon, Jun 25th, 2018 at 2:52 PM, Lucas Wang <lucasatu...@gmail.com> > wrote: > > > > > > > > > Hi Harsha, > > > > If I understand correctly, the replication quota mechanism proposed in > > KIP-73 can be helpful in that scenario. > > Have you tried it out? > > > > Thanks, > > Lucas > > > > > > > > On Sun, Jun 24, 2018 at 8:28 AM, Harsha < ka...@harsha.io > wrote: > > > > > Hi Lucas, > > > One more question, any thoughts on making this configurable > > > and also allowing subset of data requests to be prioritized. For > example > > > > > ,we notice in our cluster when we take out a broker and bring new one > it > > > > > will try to become follower and have lot of fetch requests to other > > leaders > > > in clusters. This will negatively effect the application/client > > requests. > > > We are also exploring the similar solution to de-prioritize if a new > > > replica comes in for fetch requests, we are ok with the replica to be > > > taking time but the leaders should prioritize the client requests. > > > > > > > > > Thanks, > > > Harsha > > > > > > On Fri, Jun 22nd, 2018 at 11:35 AM Lucas Wang wrote: > > > > > > > > > > > > > > > > > > > Hi Eno, > > > > > > > > Sorry for the delayed response. > > > > - I haven't implemented the feature yet, so no experimental results > so > > > > > > far. > > > > And I plan to test in out in the following days. > > > > > > > > - You are absolutely right that the priority queue does not > completely > > > > > > prevent > > > > data requests being processed ahead of controller requests. > > > > That being said, I expect it to greatly mitigate the effect of stable > > > > metadata. > > > > In any case, I'll try it out and post the results when I have it. > > > > > > > > Regards, > > > > Lucas > > > > > > > > On Wed, Jun 20, 2018 at 5:44 AM, Eno Thereska < > eno.there...@gmail.com > > > > > > > wrote: > > > > > > > > > Hi Lucas, > > > > > > > > > > Sorry for the delay, just had a look at this. A couple of > questions: > > > > > > > - did you notice any positive change after implementing this KIP? > > I'm > > > > > wondering if you have any experimental results that show the > benefit > > of > > > > the > > > > > two queues. > > > > > > > > > > - priority is usually not sufficient in addressing the problem the > > KIP > > > > > identifies. Even with priority queues, you will sometimes (often?) > > have > > > > the > > > > > case that data plane requests will be ahead of the control plane > > > > requests. > > > > > This happens because the system might have already started > > processing > > > > the > > > > > data plane requests before the control plane ones arrived. So it > > would > > > > be > > > > > good to know what % of the problem this KIP addresses. > > > > > > > > > > Thanks > > > > > Eno > > > > > > > > > > > > > > > > > > On Fri, Jun 15, 2018 at 4:44 PM, Ted Yu < yuzhih...@gmail.com > > > wrote: > > > > > > > > > > > Change looks good. > > > > > > > > > > > > Thanks > > > > > > > > > > > > On Fri, Jun 15, 2018 at 8:42 AM, Lucas Wang < > lucasatu...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Ted, > > > > > > > > > > > > > > Thanks for the suggestion. I've updated the KIP. Please take > > > another > > > > > > > > > > look. > > > > > > > > > > > > > > Lucas > > > > > > > > > > > > > > On Thu, Jun 14, 2018 at 6:34 PM, Ted Yu < yuzhih...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > > > Currently in KafkaConfig.scala : > > > > > > > > > > > > > > > > val QueuedMaxRequests = 500 > > > > > > > > > > > > > > > > It would be good if you can include the default value for > this > > > > > new > > > > > > > > > > config > > > > > > > > in the KIP. > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > On Thu, Jun 14, 2018 at 4:28 PM, Lucas Wang < > > > lucasatu...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Ted, Dong > > > > > > > > > > > > > > > > > > I've updated the KIP by adding a new config, instead of > > reusing > > > > the > > > > > > > > > existing one. > > > > > > > > > Please take another look when you have time. Thanks a lot! > > > > > > > > > > > > > > > > > > Lucas > > > > > > > > > > > > > > > > > > On Thu, Jun 14, 2018 at 2:33 PM, Ted Yu < > yuzhih...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > bq. that's a waste of resource if control request rate is > > low > > > > > > > > > > > > > > > > > > > > I don't know if control request rate can get to 100,000, > > > > likely > > > > > > not. > > > > > > > > Then > > > > > > > > > > using the same bound as that for data requests seems > high. > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 13, 2018 at 10:13 PM, Lucas Wang < > > > > > > lucasatu...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Ted, > > > > > > > > > > > > > > > > > > > > > > Thanks for taking a look at this KIP. > > > > > > > > > > > Let's say today the setting of "queued.max.requests" in > > > > > cluster A > > > > > > > is > > > > > > > > > > 1000, > > > > > > > > > > > while the setting in cluster B is 100,000. > > > > > > > > > > > The 100 times difference might have indicated that > > machines > > > > in > > > > > > > > cluster > > > > > > > > > B > > > > > > > > > > > have larger memory. > > > > > > > > > > > > > > > > > > > > > > By reusing the "queued.max.requests", the > > > > controlRequestQueue > > > > > in > > > > > > > > > cluster > > > > > > > > > > B > > > > > > > > > > > automatically > > > > > > > > > > > gets a 100x capacity without explicitly bothering the > > > > > operators. > > > > > > > > > > > I understand the counter argument can be that maybe > > that's > > > a > > > > > > > > > > waste > > > > > > > of > > > > > > > > > > > resource if control request > > > > > > > > > > > rate is low and operators may want to fine tune the > > > capacity > > > > of > > > > > > the > > > > > > > > > > > controlRequestQueue. > > > > > > > > > > > > > > > > > > > > > > I'm ok with either approach, and can change it if you > or > > > > > > anyone > > > > > > > else > > > > > > > > > > feels > > > > > > > > > > > strong about adding the extra config. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Lucas > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 13, 2018 at 3:11 PM, Ted Yu < > > > yuzhih...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Lucas: > > > > > > > > > > > > Under Rejected Alternatives, #2, can you elaborate a > > bit > > > > more > > > > > > on > > > > > > > > why > > > > > > > > > > the > > > > > > > > > > > > separate config has bigger impact ? > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 13, 2018 at 2:00 PM, Dong Lin < > > > > > lindon...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hey Luca, > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP. Looks good overall. Some > > comments > > > > > below: > > > > > > > > > > > > > > > > > > > > > > > > > > - We usually specify the full mbean for the new > > metrics > > > > in > > > > > > the > > > > > > > > KIP. > > > > > > > > > > Can > > > > > > > > > > > > you > > > > > > > > > > > > > specify it in the Public Interface section similar > > to > > > > > KIP-237 > > > > > > > > > > > > > < https://cwiki.apache.org/ > > > confluence/display/KAFKA/KIP- > > > > > > > > > > > > > > > > > 237%3A+More+Controller+Health+Metrics> > > > > > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > - Maybe we could follow the same pattern as KIP-153 > > > > > > > > > > > > > < https://cwiki.apache.org/ > > > confluence/display/KAFKA/KIP- > > > > > > > > > > > > > > > > > > > 153%3A+Include+only+client+traffic+in+BytesOutPerSec+ > > > > > > metric>, > > > > > > > > > > > > > where we keep the existing sensor name > > "BytesInPerSec" > > > > and > > > > > > add > > > > > > > a > > > > > > > > > new > > > > > > > > > > > > sensor > > > > > > > > > > > > > "ReplicationBytesInPerSec", rather than replacing > > the > > > > > sensor > > > > > > > > name " > > > > > > > > > > > > > BytesInPerSec" with e.g. "ClientBytesInPerSec". > > > > > > > > > > > > > > > > > > > > > > > > > > - It seems that the KIP changes the semantics of > the > > > > > > broker > > > > > > > > config > > > > > > > > > > > > > "queued.max.requests" because the number of total > > > > requests > > > > > > > queued > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > broker will be no longer bounded by > > > > "queued.max.requests". > > > > > > This > > > > > > > > > > > probably > > > > > > > > > > > > > needs to be specified in the Public Interfaces > > section > > > > for > > > > > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Dong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 13, 2018 at 12:45 PM, Lucas Wang < > > > > > > > > > lucasatu...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Kafka experts, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I created KIP-291 to add a separate queue for > > > > controller > > > > > > > > > requests: > > > > > > > > > > > > > > https://cwiki.apache.org/ > > > confluence/display/KAFKA/KIP- > > > > > > > > > 291% > > > > > > > > > > > > > > 3A+Have+separate+queues+for+ > > > control+requests+and+data+ > > > > > > > > > > > requests > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you please take a look and let me know your > > > > feedback? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks a lot for your time! > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Lucas > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
