Due to the incorrect initial numbering of this KIP, discussion has moved to this thread: http://mail-archives.apache.org/mod_mbox/kafka-dev/201707.mbox/%3CCAMd5Ysy3bY7Fq2xA3sk6BWW6%3D9TjT4%2Bya7mufRf6Wgre-S-UPg%40mail.gmail.com%3E
On 20 July 2017 at 16:47, Ismael Juma <ism...@juma.me.uk> wrote: > Hi Tom, > > Yes, a poll-based approach sounds good since reassignment can take a long > time. I even think that it should be manual. That is, the user should run a > command to ask for the status of the rebalance instead of the tool doing it > automatically. > > Ismael > > On Thu, Jul 20, 2017 at 4:12 AM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > > Hi Ismael, > > > > I've been working on the progress reporting assuming that it would be > > acceptable for the ReassignPartitionsCommand to poll the AdminClient API > > (and in turn the AdminClient API to poll the broker) in order to report > > progress in an interactive way. > > > > The alternative would, of course, be for the broker to push notify the > > AdminClient when it became aware of changes in progress. I didn't think > > this is what you meant as it would be a departure from the Kafka norm of > > request-response, but thought it worthwhile to check before I spend any > > more time on a polling-based approach. > > > > Thanks, > > > > Tom > > > > > > > > On 19 July 2017 at 16:08, Tom Bentley <t.j.bent...@gmail.com> wrote: > > > > > Ah, thank you! I took the number from the "Next KIP Number: 178" on the > > > KIP index and didn't check the tables. So this is now KIP-179. The old > > link > > > will point you to the right place. > > > > > > On 19 July 2017 at 15:55, Ismael Juma <ism...@juma.me.uk> wrote: > > > > > >> One more thing, it looks like there is already a KIP-178: > > >> > > >> KIP-178: Size-based log directory selection strategy > > >> > > >> Ismael > > >> > > >> On Wed, Jul 19, 2017 at 7:05 AM, Tom Bentley <t.j.bent...@gmail.com> > > >> wrote: > > >> > > >> > OK, I will work on adding support for this to the KIP, with the > > >> intention > > >> > of a two part implementation. > > >> > > > >> > On 19 July 2017 at 14:59, Ismael Juma <ism...@juma.me.uk> wrote: > > >> > > > >> > > Hi Tom, > > >> > > > > >> > > It's fine for the tool not to have this functionality from the > > start. > > >> > > However, since we're adding new Kafka protocol APIs, we need to > > >> consider > > >> > > some of these details to ensure we're building towards the end > > state, > > >> if > > >> > > that makes sense. Protocol APIs are used by multiple clients, so > > >> there is > > >> > > value in thinking ahead a bit when it comes to the design. The > > >> > > implementation can often be done in stages. > > >> > > > > >> > > Does that make sense? > > >> > > > > >> > > Ismael > > >> > > > > >> > > On Wed, Jul 19, 2017 at 6:23 AM, Tom Bentley < > t.j.bent...@gmail.com > > > > > >> > > wrote: > > >> > > > > >> > > > Hi Ismael, > > >> > > > > > >> > > > Answers in-line: > > >> > > > > > >> > > > 1. Have you considered how progress would be reported? Partition > > >> > > > > reassignment can take a long time and it would be good to > have a > > >> > > > mechanism > > >> > > > > for progress reporting. > > >> > > > > > > >> > > > > > >> > > > The ReassignPartitionsCommand doesn't currently have a mechanism > > to > > >> > track > > >> > > > progress. All you can do at the moment is initiate a > reassignment > > >> (with > > >> > > > --execute), and later check whether the assignment is in the > state > > >> you > > >> > > > asked for (with --verify). I agree it would be nice to be able > to > > >> track > > >> > > > progress. > > >> > > > > > >> > > > This will be the first 'big' bit of work I've done on Kafka, so > I > > >> would > > >> > > > prefer to limit the scope of this KIP where possible. That > said, I > > >> > > suppose > > >> > > > it could be done by having receiving controllers publish their > > >> progress > > >> > > to > > >> > > > ZooKeeper, and adding Protocol and AdminClient API for getting > > this > > >> > > > information. If you're keen on this I can certainly modify the > KIP > > >> to > > >> > add > > >> > > > this. > > >> > > > > > >> > > > Alternatively I could write a second KIP to add this ability. > What > > >> > other > > >> > > > long running tasks are there for which we'd like the ability to > > >> report > > >> > > > progress? If there are others it might be possible to come up > > with a > > >> > > common > > >> > > > mechanism. > > >> > > > > > >> > > > > > >> > > > > 2. Removals can only happen in major releases. In your > example, > > >> the > > >> > > > removal > > >> > > > > could only happen in 2.0.0. > > >> > > > > > > >> > > > > > >> > > > OK, I'll update the KIP. > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
