Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Tom, on the implications you are referring to. For me they seem the same for the __consumer_offsets and the __transaction_state topic. So I am wondering if we can rely on the same solutions for them, like providing a *.replication.factor config option. Best regards, Steven Op ma 19 mrt.

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Last week I was able to spend a bit of time working on KIP-236 again and, based on the discussion about that with Jun back in December, I refactored the controller to store the reassignment state in /brokers/topics/${topic} instead of introducing new ZK nodes. This morning I was wondering what to

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

What if we had an internal topic which watchers could listen to for information about partition reassignments? The information could be in JSON, so if we want to add new fields later, we always could. This avoids introducing a new AdminClient API. For clients that want to be notified about

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Hi Steven, I must admit that I didn't really considered that option. I can see how attractive it is from your perspective. In practice it would come with lots of edge cases which would need to be thought through: 1. What happens if the controller can't produce a record to this topic because the

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Hello Tom, when you were working out KIP-236, did you consider migrating the reassignment state from zookeeper to an internal kafka topic, keyed by partition and log compacted? It would allow an admin client and controller to easily subscribe for those changes, without the need to extend the

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

I've removed the option to pass a null reassignments argument to AdminClient.describeReassignments(), because to support this would require passing the topic and partition of each reassignment in the response, for what might be many partitions. It just seems like unnecessary bloat.

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Hi Ted, For class Reassignment, it seems you forgot to include set of brokers. I omitted a set of brokers intentionally, because the Reassignment is a an immutable reference to a (mutable) reassignment (in ZooKeeper). See KIP-236 to understand the background. I've fixed the other two issues,

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Hi Steven, I think it would be useful to be able to subscribe yourself on updates of > reassignment changes. I agree this would be really useful, but, to the extent I understand the networking underpinnings of the admin client, it might be difficult to do well in practice. Part of the problem

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Please create corresponding JIRA. For class Reassignment, it seems you forgot to include set of brokers. For class DescribeReassignmentsResult: public KafkaFuture reassignments(); the return value should be a Collection. On Fri, Dec 15, 2017 at 10:16 AM, Tom Bentley

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Tom, I think it would be useful to be able to subscribe yourself on updates of reassignment changes. Our internal kafka supervisor and monitoring tools are currently subscribed to these changes in zookeeper so they can babysit our clusters. I think it would be nice if we could receive these

[DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Hi, KIP-236 lays the foundations for AdminClient APIs to do with partition reassignment. I'd now like to start discussing KIP-240, which adds APIs to the AdminClient to list and describe the current reassignments.