Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Thank you all for your comments. As all the comments in the thread are addressed, I am starting a Vote thread for the KIP. Please have a look. Regards. Abhijeet. On Thu, Apr 25, 2024 at 6:08 PM Luke Chen wrote: > Hi, Abhijeet, > > Thanks for the update. > > I have no more comments. > > Luke

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi, Abhijeet, Thanks for the update. I have no more comments. Luke On Thu, Apr 25, 2024 at 4:21 AM Jun Rao wrote: > Hi, Abhijeet, > > Thanks for the updated KIP. It looks good to me. > > Jun > > On Mon, Apr 22, 2024 at 12:08 PM Abhijeet Kumar < > abhijeet.cse@gmail.com> > wrote: > > > Hi

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi, Abhijeet, Thanks for the updated KIP. It looks good to me. Jun On Mon, Apr 22, 2024 at 12:08 PM Abhijeet Kumar wrote: > Hi Jun, > > Please find my comments inline. > > > On Thu, Apr 18, 2024 at 3:26 AM Jun Rao wrote: > > > Hi, Abhijeet, > > > > Thanks for the reply. > > > > 1. I am

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi Jun, Please find my comments inline. On Thu, Apr 18, 2024 at 3:26 AM Jun Rao wrote: > Hi, Abhijeet, > > Thanks for the reply. > > 1. I am wondering if we could achieve the same result by just lowering > local.retention.ms and local.retention.bytes. This also allows the newly > started

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi Luke, Thanks for your comments. Please find my responses inline. On Tue, Apr 9, 2024 at 2:08 PM Luke Chen wrote: > Hi Abhijeet, > > Thanks for the KIP to improve the tiered storage feature! > > Questions: > 1. We could also get the "pending-upload-offset" and epoch via remote log >

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi, Abhijeet, Thanks for the reply. 1. I am wondering if we could achieve the same result by just lowering local.retention.ms and local.retention.bytes. This also allows the newly started follower to build up the local data before serving the consumer traffic. 2. Have you updated the KIP?

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

+1 to Jun for adding the consumer fetching from a follower scenario also to the existing section that talked about the drawback when a node built with last-tiered-offset has become a leader. As Abhijeet mentioned, we plan to have a follow-up KIP that will address these by having a deprioritzation

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi Abhijeet, Thanks for the KIP to improve the tiered storage feature! Questions: 1. We could also get the "pending-upload-offset" and epoch via remote log metadata, instead of adding a new API to fetch from the leader. Could you explain why you choose the later approach, instead of the former?

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi Christo, Please find my comments inline. On Fri, Apr 5, 2024 at 12:36 PM Christo Lolov wrote: > Hello Abhijeet and Jun, > > I have been mulling this KIP over a bit more in recent days! > > re: Jun > > I wasn't aware we apply 2.1 and 2.2 for reserving new timestamps - in > retrospect it

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi Jun, Thank you for taking the time to review the KIP. Please find my comments inline. On Fri, Apr 5, 2024 at 12:09 AM Jun Rao wrote: > Hi, Abhijeet, > > Thanks for the KIP. Left a few comments. > > 1. "A drawback of using the last-tiered-offset is that this new follower > would possess only

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hello Abhijeet and Jun, I have been mulling this KIP over a bit more in recent days! re: Jun I wasn't aware we apply 2.1 and 2.2 for reserving new timestamps - in retrospect it should have been fairly obvious. I would need to go an update KIP-1005 myself then, thank you for giving the useful

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi, Abhijeet, Thanks for the KIP. Left a few comments. 1. "A drawback of using the last-tiered-offset is that this new follower would possess only a limited number of locally stored segments. Should it ascend to the role of leader, there is a risk of needing to fetch these segments from the

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi Christo, Thanks for reviewing the KIP. The follower needs the earliest-pending-upload-offset (and the corresponding leader epoch) from the leader. This is the first offset the follower will have locally. Regards, Abhijeet. On Fri, Mar 29, 2024 at 1:14 PM Christo Lolov wrote: > Heya! > >

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Heya! First of all, thank you very much for the proposal, you have explained the problem you want solved very well - I think a faster bootstrap of an empty replica is definitely an improvement! For my understanding, which concrete offset do you want the leader to give back to a follower -

[DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi All, I have created KIP-1023 to introduce follower fetch from tiered offset. This feature will be helpful in significantly reducing Kafka rebalance/rebuild times when the cluster is enabled with tiered storage.