+ Leigh Stewart (for the connection concern on current proposal) Rithin,
Leigh, Yiming and me had a brief discussion at lunch. Leigh raised the idea of using P2C load balancing algorithm[1] (to avoid talking to all bookies). The idea has been implemented and described in finagle load balancing [2]. It would be good to discuss and think if we can use this approach to avoid pulling all bookies. Leigh, if you can, can you describe more about the idea of using P2C load balancing. [1] Michael Mitzenmacher. 2001. The Power of Two Choices in Randomized Load Balancing. IEEE Trans. Parallel Distrib. [2] Finagle Load Balancing. https://twitter.github.io/finagle/guide/Clients.html#power-of-two-choices-p2c-least-loaded - Sijie On Tue, Dec 13, 2016 at 7:12 PM, Sijie Guo <[email protected]> wrote: > Hi Rithin, > > The proposal looks good to me. I have a few comments: > > - How will the BookieInfo structure look like? > - I like the idea of a general GET_BOOKIE_INFO request. Do you have any > proposed wire protocol changes? > - I assume that you might need the clients to configure a function to > compute the weight for a bookie, right? So we can also compute the weight > based on network bandwidth? > - How does bookie collect the resource information? > > - Sijie > > > On Mon, Dec 12, 2016 at 11:09 AM, Rithin Shetty <[email protected]> wrote: > >> Hi Sijie, >> >> I've updated the proposal page now: >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-2+ >> -+Resource+aware+data+placement. >> Sorry about the delay. >> >> Thanks, >> --Rithin >> >> On Sat, Dec 10, 2016 at 1:45 AM, Sijie Guo <[email protected]> wrote: >> >> > As a reminder, we will be discussing the resource aware placement at the >> > meeting next week. >> > >> > Rithin, do you mind updating the proposal by next Tue? so we have time >> to >> > check before the meeting. >> > >> > - Sijie >> > >> > On Tue, Nov 1, 2016 at 2:29 AM, Sijie Guo <[email protected]> wrote: >> > >> > > https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-2+ >> > > -+Resource+aware+data+placement >> > > >> > > I created a wiki page for place holder. Anyone that is interested in >> this >> > > topic can help fill a basic proposal and drive the discussion? >> > > >> > > *Problem* >> > > >> > > There are bunch of tickets around talking about placement policy : >> > > - handle different types of storage (tier storage) >> > > - handle storage node having different type of resources (resource >> aware) >> > > - better to canary/bucket-testing new version (manageability) >> > > >> > > It would be good to consolidate these thoughts to provide a common >> > > framework/solution to meet those basic requirements. >> > > >> > > *Proposal* >> > > >> > > [TBD] >> > > >> > > - Sijie >> > > >> > > >> > >> > >
