My current interpretation is that if I start a partition reassignment, for the sake of simplicity let's assume it's just for a single partition, the new leader will first become a follower of the current leader, and when it has caught up it'll transfer leadership over to itself?
Partition reassignment will not move the leader unless the old leader is not part of the new set of replicas. Even when it does move the leader, it waits until the new replicas enter the ISR. If so, is Kafka only concerned about data that has been flushed to disk? So if there's data that hasn't been flushed to disk yet it'll be discarded when the new leader gains leadership? No. Kafka only waits until a replica enters the ISR set for it to be a candidate for leadership. What happens if neither of the new ARs are part of the previous AR set? I suppose that they'll all be followers and when the new leader has caught up it'll assume leadership? Yes, this is the only time the leader is moved as part of partition reassignment. I've played around with controlled shutdown and noticed that the broker that's shutting down is truncating logs prior to actually shutting down, which I don't really understand? can you share the logs? Is this truncation during the shutdown? On Sun, Sep 21, 2014 at 2:32 PM, Mathias Söderberg < mathias.soederb...@gmail.com> wrote: > Good day, > > We're currently running a Kafka cluster in our staging environment, testing > everything out and so far it runs fairly smoothly. > > One thing that we're currently looking into is scaling a cluster that is > already up 'n serving data, and I have some questions regarding moving > partitions between brokers. > > My current interpretation is that if I start a partition reassignment, for > the sake of simplicity let's assume it's just for a single partition, the > new leader will first become a follower of the current leader, and when it > has caught up it'll transfer leadership over to itself? > If so, is Kafka only concerned about data that has been flushed to disk? So > if there's data that hasn't been flushed to disk yet it'll be discarded > when the new leader gains leadership? > > What happens if neither of the new ARs are part of the previous AR set? I > suppose that they'll all be followers and when the new leader has caught up > it'll assume leadership? > > I've played around with controlled shutdown and noticed that the broker > that's shutting down is truncating logs prior to actually shutting down, > which I don't really understand? I cannot really tell whether it does this > for all topics or just a subset of the ones that it's AR for as we have > quite a lot of topics, but I figured I'd just throw it out there, as I feel > like it could be explained by the answers to the questions above. > > Should mention that we're running v0.8.1.1 with mostly stock configuration, > and that our producers are currently using ACK = 1 as well. > > Best regards, > Mathias >
