Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Thanks, Sergio. On Fri, Mar 25, 2022 at 7:41 AM Sergio Daniel Troiano wrote: > > Hi David, > > I apologize. I missed your suggestion. > By the way I like it and I have applied your suggestion. > > About the rejected alternatives I have updated the KIP as well > > Best regards > Sergio Troiano > >

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi David, I apologize. I missed your suggestion. By the way I like it and I have applied your suggestion. About the rejected alternatives I have updated the KIP as well Best regards Sergio Troiano On Fri, 25 Mar 2022 at 06:50, David Jacot wrote: > Hi Sergio, > > I made a suggestion a few week

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Sergio, I made a suggestion a few weeks ago about the name about the parameter but haven’t got a response for it. Did you consider it? Do we need to update the rejected alternatives section to mention the alternative options discussed in this thread? Thanks, David Le ven. 25 mars 2022 à 03:4

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Sergio, Thanks for asking. Since it's been discussed for weeks, you can start the vote anytime. Thank you. Luke On Fri, Mar 25, 2022 at 10:40 AM Sergio Troiano wrote: > Hey guys, > > > What is the next step? Who decides when it is time for voting? > > > Thanks! > > Sent from my iPhone > > >

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hey guys, What is the next step? Who decides when it is time for voting? Thanks! Sent from my iPhone > On 8 Mar 2022, at 19:57, Sergio Daniel Troiano > wrote: > >  > Hi Michael, > > Yes, it's a good idea and I considered it, the main problem is the > FileRecords class does not accept nu

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Michael, Yes, it's a good idea and I considered it, the main problem is the *FileRecords* class does not accept number of batches as a parameter, it accepts bytes instead, so if we want to do so either we redesign a core class or we create a new one. One of the pretty things (I consider) about

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Sergio, Thanks for the KIP. Instead of specifying the size in bytes, have you considered specifying it in terms of number of batches? I think it's a bit more user friendly than a size in raw bytes. For example: --num-batches: The number of batches to read from the log segment. Thanks, Mickael

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Luke, Make sense, done! Thank you. Sergio Troiano On Tue, 8 Mar 2022 at 03:02, Luke Chen wrote: > Hi Sergio, > > > I don't want this to minimize the main feature I want to deploy as I > think the > message size limit is not as important as the limiting the amount of > batches. > > Agree! Le

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Sergio, > I don't want this to minimize the main feature I want to deploy as I think the message size limit is not as important as the limiting the amount of batches. Agree! Let's focus on the feature of limiting the batch amounts. One more comment to the KIP: 1. Could you put the new paramet

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

hey Luke, I am interested in expanding the KIP scope but I am a bit concerned this could create a lot of noise and confusion as they look like very similar parameters, I agree this is a small change, so I think if I do it properly it should not be a problem at all, I just will need a couple more o

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Sergio, Thanks for your explanation. Make sense to me. > Only interesting thing that I have just found is *max-message-size *is not used while dump logs are requested, instead it is used by dumpIndex Are you interested in expanding the scope of this KIP to include the *max-message-size* in du

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

hey Luke, Let me answer them: 1. If the *max-batches-size* is too small that results in no records output, will we output any information to the user? If the *max-batches-size*is even smaller than the first batch then there won't be any output, this is handled by FileRecords class, I think this

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

hey Kirk, Thanks for the questions, please let me answer them: 1. This is handled by the *FileRecords class, *now the open uses the slice which takes care of the end

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Sergio, I wonder if « max-bytes » would be a better name than « max-batches-size ». The intend is more explicit. What do you think? Best, David Le sam. 5 mars 2022 à 10:36, Luke Chen a écrit : > Hi Sergio, > > Thanks for the explanation! Very clear! > I think we should put this example and

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Sergio, Thanks for the explanation! Very clear! I think we should put this example and explanation into KIP. Other comments: 1. If the *max-batches-size* is too small that results in no records output, will we output any information to the user? 2. After your explanation, I guess the use of *m

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

hey Luke, thanks for the interest, it is a good question, please let me explain you: *max-message-size *a filter for the size of each batch, so for example if Iset --max-message-size 1000 bytes and my segment log has 300 batches, 150 of them has a size of 500 bytes and the other 150 has a size o

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Sergio, Thanks for the KIP! One question: I saw there's a `max-message-size` argument that seems to do the same thing as you want. Could you help explain what's the difference between `max-message-size` and `max-batches-size`? Thank you. Luke On Sat, Mar 5, 2022 at 3:21 AM Kirk True wrote:

Re: [DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

Hi Sergio, Thanks for the KIP. I don't know anything about the log segment internals, but the logic and implementation seem sound. Three questions: 1. Since the --max-batches-size unit is bytes, does it matter if that size doesn't align to a record boundary? 2. Can you add a check to make sur

[DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

https://cwiki.apache.org/confluence/display/KAFKA/KIP-824%3A+Allowing+dumping+segmentlogs+limiting+the+batches+in+the+output

[DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

https://cwiki.apache.org/confluence/display/KAFKA/KIP-824%3A+Allowing+dumping+segmentlogs+limiting+the+batches+in+the+output