Hi Tom, Yes, a poll-based approach sounds good since reassignment can take a long time. I even think that it should be manual. That is, the user should run a command to ask for the status of the rebalance instead of the tool doing it automatically.
Ismael On Thu, Jul 20, 2017 at 4:12 AM, Tom Bentley <t.j.bent...@gmail.com> wrote: > Hi Ismael, > > I've been working on the progress reporting assuming that it would be > acceptable for the ReassignPartitionsCommand to poll the AdminClient API > (and in turn the AdminClient API to poll the broker) in order to report > progress in an interactive way. > > The alternative would, of course, be for the broker to push notify the > AdminClient when it became aware of changes in progress. I didn't think > this is what you meant as it would be a departure from the Kafka norm of > request-response, but thought it worthwhile to check before I spend any > more time on a polling-based approach. > > Thanks, > > Tom > > > > On 19 July 2017 at 16:08, Tom Bentley <t.j.bent...@gmail.com> wrote: > > > Ah, thank you! I took the number from the "Next KIP Number: 178" on the > > KIP index and didn't check the tables. So this is now KIP-179. The old > link > > will point you to the right place. > > > > On 19 July 2017 at 15:55, Ismael Juma <ism...@juma.me.uk> wrote: > > > >> One more thing, it looks like there is already a KIP-178: > >> > >> KIP-178: Size-based log directory selection strategy > >> > >> Ismael > >> > >> On Wed, Jul 19, 2017 at 7:05 AM, Tom Bentley <t.j.bent...@gmail.com> > >> wrote: > >> > >> > OK, I will work on adding support for this to the KIP, with the > >> intention > >> > of a two part implementation. > >> > > >> > On 19 July 2017 at 14:59, Ismael Juma <ism...@juma.me.uk> wrote: > >> > > >> > > Hi Tom, > >> > > > >> > > It's fine for the tool not to have this functionality from the > start. > >> > > However, since we're adding new Kafka protocol APIs, we need to > >> consider > >> > > some of these details to ensure we're building towards the end > state, > >> if > >> > > that makes sense. Protocol APIs are used by multiple clients, so > >> there is > >> > > value in thinking ahead a bit when it comes to the design. The > >> > > implementation can often be done in stages. > >> > > > >> > > Does that make sense? > >> > > > >> > > Ismael > >> > > > >> > > On Wed, Jul 19, 2017 at 6:23 AM, Tom Bentley <t.j.bent...@gmail.com > > > >> > > wrote: > >> > > > >> > > > Hi Ismael, > >> > > > > >> > > > Answers in-line: > >> > > > > >> > > > 1. Have you considered how progress would be reported? Partition > >> > > > > reassignment can take a long time and it would be good to have a > >> > > > mechanism > >> > > > > for progress reporting. > >> > > > > > >> > > > > >> > > > The ReassignPartitionsCommand doesn't currently have a mechanism > to > >> > track > >> > > > progress. All you can do at the moment is initiate a reassignment > >> (with > >> > > > --execute), and later check whether the assignment is in the state > >> you > >> > > > asked for (with --verify). I agree it would be nice to be able to > >> track > >> > > > progress. > >> > > > > >> > > > This will be the first 'big' bit of work I've done on Kafka, so I > >> would > >> > > > prefer to limit the scope of this KIP where possible. That said, I > >> > > suppose > >> > > > it could be done by having receiving controllers publish their > >> progress > >> > > to > >> > > > ZooKeeper, and adding Protocol and AdminClient API for getting > this > >> > > > information. If you're keen on this I can certainly modify the KIP > >> to > >> > add > >> > > > this. > >> > > > > >> > > > Alternatively I could write a second KIP to add this ability. What > >> > other > >> > > > long running tasks are there for which we'd like the ability to > >> report > >> > > > progress? If there are others it might be possible to come up > with a > >> > > common > >> > > > mechanism. > >> > > > > >> > > > > >> > > > > 2. Removals can only happen in major releases. In your example, > >> the > >> > > > removal > >> > > > > could only happen in 2.0.0. > >> > > > > > >> > > > > >> > > > OK, I'll update the KIP. > >> > > > > >> > > > >> > > >> > > > > >
