hi Andrew Yes, we are on the same page :)
Best, Chia-Ping > Andrew Schofield <[email protected]> 於 2026年7月8日 晚上10:47 寫道： > > Hi, > I'd like to confirm what is happening with this KIP for AK 4.4. I think the > answer is that we've decided to improve the docs (I see a PR) and leave it at > that for now. Do I understand correctly? > > Thanks, > Andrew > >> On 2026/07/08 01:32:41 Chia-Ping Tsai wrote: >> hi all, >> >> While I agree we shouldn't surprise the majority of users, relying solely on >> documentation means some users will still fall into this trap. Realizing >> this limitation only when data is silently lost during a partition expansion >> is a truly frustrating experience. >> >> Since we've reached the KIP freeze for 4.4.0, I recommend we focus on >> updating the docs first. >> >> We can put this KIP on hold for now and revisit it after Community Over >> Code. IIRC, Jiunn-Yang is going to share this specific issue at the event >> with other Kafka users and providers. Hopefully, we can gather some fresh >> perspectives and inspiration from the community there. >> >> Best, >> Chia-Ping >> >>> On 2026/07/07 21:38:22 Jun Rao via dev wrote: >>> Hi, Jiunn-Yang, >>> >>> Thanks for the reply. >>> >>> Changing the default value only makes sense if it benefits most users. >>> Most users >>> probably never increase the number of partitions. Changing the default >>> config value gives them no benefit and only introduces a probably >>> surprising behavior >>> change. >>> >>> Jun >>> >>>> On Tue, Jul 7, 2026 at 1:22 PM Chia-Ping Tsai <[email protected]> wrote: >>> >>>> hi Jiunn >>>> >>>> +1 to Approach A :) >>>> >>>> Best, >>>> Chia-Ping >>>> >>>> On 2026/07/07 11:55:14 黃竣陽 wrote: >>>>> Hi Jun, Chia-Ping, >>>>> >>>>> I'd like to first explain why I believe changing the default is >>>> worthwhile. >>>>> >>>>> Documentation can only help users who already know about the problem. >>>>> Because the failure mode is silent, most users discover it only after >>>> data has >>>>> already been missed. A safer default such as by_duration:PT5S protects >>>> users >>>>> automatically, without requiring prior awareness of the issue. >>>>> >>>>> Given that, the question is not whether the issue exists, but how best >>>> to introduce >>>>> a safer default. >>>>> >>>>> Approach A — Follow a deprecation cycle and land the change in 5.0 >>>>> - Keep the current default (`latest`) in upcoming 4.x releases, but emit >>>> a warning that >>>>> the default will change to `by_duration:PT5S` in 5.0 and explain how >>>> users can retain >>>>> the current behavior explicitly. >>>>> - Change the default to `by_duration:PT5S` in 5.0. >>>>> - Update shell tools (for example, the console consumer) to explicitly >>>> use `latest`, preserving >>>>> the expected "show only new messages" behavior. >>>>> >>>>> Approach B — Change the default in a 4.4 minor release >>>>> - Change the default to `by_duration:PT5S` in a 4.x minor release. >>>>> - Leave shell tools unchanged. >>>>> - Emit a warning explaining the change and how users can explicitly >>>> restore `latest`. >>>>> >>>>> I favor Approach B: >>>>> >>>>> Silent data loss during partition expansion is a correctness issue, not >>>> a usability issue. >>>>> Delaying the safer default until 5.0 leaves users relying on the >>>> implicit default exposed unnecessarily. >>>>> Users who need latest can simply pin it explicitly. >>>>> I don't think shell tools need special handling. by_duration:PT5S only >>>> replays a few seconds of data, >>>>> which is negligible in practice and preferable to silently missing >>>> records. Keeping the behavior consistent >>>>> is simpler. >>>>> >>>>> Best Regards, >>>>> Jiunn-Yang >>>>> >>>>>> Jun Rao via dev <[email protected]> 於 2026年7月7日 清晨5:49 寫道： >>>>>> >>>>>> Hi, Chia-Ping, >>>>>> >>>>>> I am not sure that we need an out-of-the-box solution. The proposal in >>>> the >>>>>> KIP is also not out-of-the-box and requires users to opt-in. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Jun >>>>>> >>>>>> On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <[email protected]> >>>> wrote: >>>>>> >>>>>>> hi Jun >>>>>>> >>>>>>> Yes, updating the docs requires minimal changes to the codebase, but >>>> it is >>>>>>> not an out-of-the-box solution. From my perspective, this issue can be >>>>>>> addressed by setting the console consumer's policy back to latest. >>>>>>> >>>>>>> Let's see what Jiunn thinks >>>>>>> >>>>>>> Best, >>>>>>> Chia-Ping >>>>>>> >>>>>>> On 2026/07/06 15:59:37 Jun Rao via dev wrote: >>>>>>>> Hi, Chia-Ping, >>>>>>>> >>>>>>>> One potential downside of changing the default is that some existing >>>>>>> tools >>>>>>>> (e.g. console consumer) may depend on this behavior. For example, a >>>> user >>>>>>>> testing the system may expect to see only new messages and could be >>>>>>>> surprised by the change. Another option is to leave the default >>>>>>> unchanged, >>>>>>>> but document clearly how to prevent losing messages on new partitions >>>>>>>> through by_duration. The users can then make the corresponding config >>>>>>>> changes if needed. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Jun >>>>>>>> >>>>>>>> On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <[email protected]> >>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Jun, >>>>>>>>> >>>>>>>>> Thanks for the feedback. I agree with your conclusion. It would be >>>>>>> better >>>>>>>>> to "make haste" with a new KIP to address this. We can focus the >>>>>>> discussion >>>>>>>>> on the following points: >>>>>>>>> >>>>>>>>> 1) Should we change the default reset policy to by_duration for all >>>>>>>>> consumers? >>>>>>>>> 2) What is the optimal duration to prevent data loss while avoiding >>>>>>> large >>>>>>>>> backlogs? Note that the classic consumer has a 3-second heartbeat >>>>>>> interval, >>>>>>>>> whereas the async consumer uses 5 seconds. >>>>>>>>> 3) What are the potential side effects of replacing latest with >>>>>>>>> by_duration as the default behavior? >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Chia-Ping >>>>>>>>> >>>>>>>>> On 2026/06/30 20:39:24 Jun Rao via dev wrote: >>>>>>>>>> Hi, Chia-Ping, >>>>>>>>>> >>>>>>>>>> I agree that using by_duration with a value of 5s to 10s addresses >>>>>>> most >>>>>>>>>> common issues with expanded partitions, given the default >>>>>>>>>> heartbeat.interval.ms is 3 seconds. We can consider setting that >>>> as >>>>>>> the >>>>>>>>> new >>>>>>>>>> default. The current proposal may address some other rarer cases, >>>>>>> but it >>>>>>>>>> feels a bit like over engineering. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Jun >>>>>>>>>> >>>>>>>>>> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai < >>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> - Short durations fail during extended downtimes, leading to the >>>>>>> exact >>>>>>>>>>> silent data >>>>>>>>>>> loss this KIP aims to prevent. >>>>>>>>>>> >>>>>>>>>>> Losing the records produced to expanded partitions during a >>>>>>> downtime is >>>>>>>>>>> actually the expected behavior for the latest policy. This is >>>>>>> exactly >>>>>>>>> why, >>>>>>>>>>> in the current KIP proposal, we prefer by_duration over earliest >>>>>>> for >>>>>>>>> the >>>>>>>>>>> expanded partitions—we want to avoid forcing users to consume a >>>>>>> massive >>>>>>>>>>> accumulated backlog just because a partition is technically "new". >>>>>>>>>>> >>>>>>>>>>> To quote my previous comment from May 15: >>>>>>>>>>> "...if a consumer is offline for a while and a new partition is >>>>>>> created >>>>>>>>>>> during that downtime, the user might actually want to skip to >>>>>>> latest >>>>>>>>> when >>>>>>>>>>> resuming, rather than reading from earliest just because the >>>>>>> partition >>>>>>>>> is >>>>>>>>>>> technically 'new' to the group." >>>>>>>>>>> >>>>>>>>>>> On 2026/06/30 12:12:47 黃竣陽 wrote: >>>>>>>>>>>> Hello all, >>>>>>>>>>>> >>>>>>>>>>>> The metadata blindness and consumer downtime windows are >>>>>>> inherently >>>>>>>>>>>> unpredictable, making a fixed by_duration approach unviable. No >>>>>>>>> static >>>>>>>>>>>> duration can accommodate both extremes: >>>>>>>>>>>> >>>>>>>>>>>> - Short durations fail during extended downtimes, leading to the >>>>>>>>> exact >>>>>>>>>>> silent data >>>>>>>>>>>> loss this KIP aims to prevent. >>>>>>>>>>>> - Long durations (calibrated for worst-case scenarios) cause >>>>>>> routine >>>>>>>>>>> restarts to >>>>>>>>>>>> trigger unnecessary mass replays, undermining the latest offset >>>>>>> reset >>>>>>>>>>> policy. >>>>>>>>>>>> >>>>>>>>>>>> This is why the proposed solution uses creation time. It >>>>>>> provides a >>>>>>>>>>> discrete, >>>>>>>>>>>> deterministic signal—verifying whether a partition is actually >>>>>>> newer >>>>>>>>>>> than the group, >>>>>>>>>>>> rather than relying on an arbitrary time window that has no >>>>>>>>> correlation >>>>>>>>>>> with the consumer’s >>>>>>>>>>>> actual downtime. >>>>>>>>>>>> >>>>>>>>>>>> Best Regards, >>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>> >>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月30日 下午3:05 寫道： >>>>>>>>>>>>> >>>>>>>>>>>>> Hi all, >>>>>>>>>>>>> >>>>>>>>>>>>> I'd like to revisit Jun previous comment about using >>>>>>>>> by_duration=5s. I >>>>>>>>>>> realize now that having a small backlog is actually an acceptable >>>>>>>>>>> trade-off, especially since the latest policy inherently brings a >>>>>>> tiny >>>>>>>>>>> backlog to starting consumers anyway due to the time gap between >>>>>>>>> fetching >>>>>>>>>>> the offset and actually consuming. >>>>>>>>>>>>> >>>>>>>>>>>>> If we bring the simple by_duration=5s solution back to the >>>>>>> KIP, I'd >>>>>>>>>>> like to discuss a further step: Should we just change the default >>>>>>>>> policy >>>>>>>>>>> from latest to by_duration=5s specifically for the new consumer >>>> and >>>>>>>>> share >>>>>>>>>>> consumer? >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>> >>>>>>>>>>>>> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote: >>>>>>>>>>>>>> hi Jun >>>>>>>>>>>>>> >>>>>>>>>>>>>> The most important part of this story is how users should >>>>>>> expect >>>>>>>>> the >>>>>>>>>>> data >>>>>>>>>>>>>> they can see when using the latest or by_duration policy with >>>>>>>>> expanded >>>>>>>>>>>>>> partitions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is >>>>>>>>>>>>>> non-deterministic, which means users will either read too many >>>>>>>>>>> historical >>>>>>>>>>>>>> records from existing partitions or lose some records from >>>>>>>>> expanded >>>>>>>>>>>>>> partitions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms is a bit >>>>>>> hard to >>>>>>>>>>>>>> understand, and that is why I preferred having a whole new >>>>>>> policy >>>>>>>>>>> based >>>>>>>>>>>>>> entirely on group creation time (KIP-1282) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 >>>>>>> 上午1:08寫道： >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for the reply. I am still trying to understand the >>>>>>> value >>>>>>>>> of >>>>>>>>>>> the new >>>>>>>>>>>>>>> configs with the KIP. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't want to >>>>>>> miss the >>>>>>>>>>> data if >>>>>>>>>>>>>>> the backlog is small. The backlog of the existing partition >>>>>>> is >>>>>>>>> easy >>>>>>>>>>> to >>>>>>>>>>>>>>> understand because it relates to retention time. The backlog >>>>>>> for >>>>>>>>> the >>>>>>>>>>> new >>>>>>>>>>>>>>> partition is a bit subtle to understand since it depends on >>>>>>> the >>>>>>>>>>> metadata >>>>>>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms, the user >>>>>>>>> needs >>>>>>>>>>> to >>>>>>>>>>>>>>> understand the metadata refresh delay on the consumer side >>>>>>> and >>>>>>>>> use >>>>>>>>>>> it to >>>>>>>>>>>>>>> set the config. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Now, let's consider the alternative: setting the same value >>>>>>> for >>>>>>>>> the >>>>>>>>>>>>>>> existing by_duration policy. The KIP lists three issues with >>>>>>> this >>>>>>>>>>> approach. >>>>>>>>>>>>>>> 1. It computes the seek target client-side as now() - >>>>>>> duration, >>>>>>>>> which >>>>>>>>>>>>>>> introduces clock skew across consumers and forces operators >>>>>>> to >>>>>>>>> choose >>>>>>>>>>>>>>> overly large durations, causing unnecessary reprocessing. >>>>>>>>>>>>>>> 2. The target timestamp is recomputed on each retry, so >>>>>>> failed >>>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target forward and >>>>>>>>>>> potentially >>>>>>>>>>>>>>> miss records produced between attempts. >>>>>>>>>>>>>>> 3. It applies uniformly to all partitions without committed >>>>>>>>> offsets, >>>>>>>>>>> and >>>>>>>>>>>>>>> cannot distinguish newly expanded partitions from >>>>>>> long-existing >>>>>>>>>>> partitions >>>>>>>>>>>>>>> newly assigned to the group, leading to unnecessary replay. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a >>>>>>> bit >>>>>>>>>>> buffer to >>>>>>>>>>>>>>> the metadata refresh delay. We could also consider improving >>>>>>> the >>>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh delay is >>>>>>>>> typically >>>>>>>>>>> low >>>>>>>>>>>>>>> (in the order of minutes with the classic consumer and tens >>>>>>> of >>>>>>>>>>> seconds with >>>>>>>>>>>>>>> the new consumer). If a user is ok with reading that much >>>>>>> backlog >>>>>>>>>>> for new >>>>>>>>>>>>>>> partitions, it seems they will be ok doing the same for >>>>>>> existing >>>>>>>>>>>>>>> partitions. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So, instead of introducing a new config, could we just reuse >>>>>>> the >>>>>>>>>>> existing >>>>>>>>>>>>>>> config with better documentation and/or implementation? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <[email protected]> >>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hello Jun, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> You're right that group creation time is the more intuitive >>>>>>>>> answer >>>>>>>>>>> at >>>>>>>>>>>>>>>> first glance, >>>>>>>>>>>>>>>> the KIP's own motivation talks about partitions that >>>>>>> "predate >>>>>>>>> the >>>>>>>>>>> group" >>>>>>>>>>>>>>>> vs partitions >>>>>>>>>>>>>>>> "created during group runtime," which directly points to a >>>>>>>>>>>>>>> group-lifecycle >>>>>>>>>>>>>>>> classifier. >>>>>>>>>>>>>>>> I'd like to walk through why we landed on partition age, >>>>>>> and the >>>>>>>>>>>>>>>> trade-offs we considered. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We evaluated three candidate signals: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. `by_duration:5secs` >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This covers the metadata blindness window, but has issues >>>>>>> the >>>>>>>>> KIP >>>>>>>>>>>>>>>> currently documents >>>>>>>>>>>>>>>> under "Why not use `by_duration`?": >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock skew >>>>>>> across >>>>>>>>>>> consumers. >>>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward, >>>>>>> potentially >>>>>>>>>>> missing >>>>>>>>>>>>>>>> records produced between >>>>>>>>>>>>>>>> attempts. >>>>>>>>>>>>>>>> - It applies uniformly to all partitions without committed >>>>>>>>> offsets, >>>>>>>>>>>>>>>> including pre-existing partitions >>>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary replay. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. Group creation time as classifier >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This works cleanly when the consumer is actively running. >>>>>>> Our >>>>>>>>>>> concern >>>>>>>>>>>>>>>> is the idle / late-rejoin case: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> T=0: Group created. >>>>>>>>>>>>>>>> T=1..T=100: Consumer idle (down, disconnected, etc.). >>>>>>>>>>>>>>>> T=50: Partition added during the idle window. >>>>>>>>>>>>>>>> T=100: Consumer resumes. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Under group creation time, the new partition is classified >>>>>>> as >>>>>>>>> new >>>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything >>>>>>> from >>>>>>>>> T=50. >>>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated >>>>>>> data >>>>>>>>>>> that >>>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely the >>>>>>> contract of >>>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to >>>>>>>>> treat >>>>>>>>>>>>>>>> the new partition differently; both contain backlog >>>>>>> accumulated >>>>>>>>>>> during >>>>>>>>>>>>>>>> the same idle window. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This aligns with the "backlog is backlog” principle you >>>>>>> raised >>>>>>>>> in >>>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some >>>>>>> backlog >>>>>>>>> on >>>>>>>>>>>>>>>> every other partition during the same idle period; forcing >>>>>>>>> 0-backlog >>>>>>>>>>>>>>>> tolerance only on new partitions would be inconsistent with >>>>>>> that >>>>>>>>>>>>>>>> tolerance. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3. Partition age vs threshold >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Partition age corresponds to the actual silent data loss >>>>>>> window, >>>>>>>>>>>>>>>> the gap between partition creation and the consumer’s >>>>>>> metadata >>>>>>>>>>>>>>>> refresh. Within this window, data loss is genuinely silent: >>>>>>> the >>>>>>>>>>>>>>>> consumer had no opportunity to know about the partition. >>>>>>> Outside >>>>>>>>>>> this >>>>>>>>>>>>>>>> window, missing data reflects either: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle >>>>>>> consumers, >>>>>>>>> or >>>>>>>>>>>>>>>> - (b) an operational issue to surface via monitoring, not >>>>>>> via >>>>>>>>> reset >>>>>>>>>>>>>>> policy. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We did not choose partition age because it is more elegant >>>>>>> than >>>>>>>>>>> group >>>>>>>>>>>>>>>> creation time — we chose it because its failure mode >>>>>>> (requires a >>>>>>>>>>>>>>>> threshold) is >>>>>>>>>>>>>>>> less invasive than the failure mode of group creation time >>>>>>>>>>> (overrides >>>>>>>>>>>>>>>> user-stated >>>>>>>>>>>>>>>> `latest` intent during idle periods). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 上午11:52 >>>>>>> 寫道： >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Jun, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Relying on both creation times will create an inconsistent >>>>>>>>>>> scenario. A >>>>>>>>>>>>>>>>> consumer that lost all offsets due to a long sleep will >>>>>>> seek >>>>>>>>> to the >>>>>>>>>>>>>>>>> beginning for the partitions created later than the group. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the >>>>>>>>> inconsistency >>>>>>>>>>>>>>>> using a >>>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a >>>>>>> consensus, >>>>>>>>>>> KIP-1327 >>>>>>>>>>>>>>>> goes >>>>>>>>>>>>>>>>> back to using flexible configurations to prevent users from >>>>>>>>> falling >>>>>>>>>>>>>>> into >>>>>>>>>>>>>>>>> that pitfall. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, Chia-Ping >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年6月13日週六 >>>>>>>>> 上午6:49寫道： >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms still >>>>>>> feels >>>>>>>>>>> weird to >>>>>>>>>>>>>>>> me. >>>>>>>>>>>>>>>>>> It >>>>>>>>>>>>>>>>>> categorizes partitions as new or existing based on the >>>>>>>>> partition >>>>>>>>>>>>>>>> creation >>>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be based on >>>>>>> the >>>>>>>>> group >>>>>>>>>>>>>>>> creation >>>>>>>>>>>>>>>>>> time: all partitions existing when the group is created >>>>>>> are >>>>>>>>>>> existing >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> all partitions created after the group creation are new >>>>>>>>>>> partitions. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <[email protected]> >>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no further >>>>>>>>>>>>>>>>>>> discussion, I will close the vote. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hello Jian, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks for your feedback, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common operational >>>>>>> task, >>>>>>>>> not an >>>>>>>>>>>>>>> edge >>>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 下午5:49 寫道： >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to >>>>>>> clarify >>>>>>>>> that >>>>>>>>>>> this >>>>>>>>>>>>>>>>>> is a >>>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case, which further >>>>>>>>>>>>>>> demonstrates >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> need for this optimization. For example: >>>>>>>>>>>>>>>>>>>>> A partition expansion is a common operational task in >>>>>>>>> Kafka: To >>>>>>>>>>>>>>>>>> balance >>>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are typically >>>>>>> created >>>>>>>>>>> with a >>>>>>>>>>>>>>>>>>> moderate >>>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic grows over >>>>>>>>> time, >>>>>>>>>>> it is >>>>>>>>>>>>>>>>>>> often >>>>>>>>>>>>>>>>>>>>> necessary to increase the number of partitions to >>>>>>>>> accommodate >>>>>>>>>>> the >>>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>>>> workload. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>>> Jian >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hello chia, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP! >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月30日 >>>>>>> 晚上8:29 >>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" >>>>>>> when >>>>>>>>>>>>>>> describing >>>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using >>>>>>> them to >>>>>>>>>>>>>>> describe >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but >>>>>>>>> applying >>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>>>> terms >>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care >>>>>>>>> whether >>>>>>>>>>> a >>>>>>>>>>>>>>>>>>> partition >>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that >>>>>>>>> they >>>>>>>>>>> won't >>>>>>>>>>>>>>>>>>> silently >>>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the topic during >>>>>>> their >>>>>>>>>>> active >>>>>>>>>>>>>>>>>>>>>> consumption. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 下午12:30寫道： >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP >>>>>>>>> motivation >>>>>>>>>>>>>>>> section. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 >>>>>>> 2026年5月30日 >>>>>>>>>>> 凌晨1:12 寫道： >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger >>>>>>>>>>> motivation for >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> KIP. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all >>>>>>>>> partitions >>>>>>>>>>>>>>> without >>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded >>>>>>>>> partition >>>>>>>>>>> (hot) >>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the >>>>>>> consumer >>>>>>>>> has >>>>>>>>>>> never >>>>>>>>>>>>>>>>>> seen >>>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the >>>>>>> hot >>>>>>>>>>> partition >>>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is >>>>>>> also >>>>>>>>>>>>>>>> insufficient >>>>>>>>>>>>>>>>>>>>>>>> because: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) >>>>>>> varies >>>>>>>>> across >>>>>>>>>>>>>>> nodes >>>>>>>>>>>>>>>>>>> due >>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly >>>>>>> large >>>>>>>>>>>>>>> duration, >>>>>>>>>>>>>>>>>>>>>>>> causing >>>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing. >>>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the >>>>>>> seek >>>>>>>>> time >>>>>>>>>>> on >>>>>>>>>>>>>>>>>> retry, >>>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking >>>>>>> data >>>>>>>>>>> loss." >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If >>>>>>> these >>>>>>>>> issues >>>>>>>>>>>>>>>>>> persist, >>>>>>>>>>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare >>>>>>> situations >>>>>>>>>>> don't >>>>>>>>>>>>>>>>>> need a >>>>>>>>>>>>>>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare >>>>>>> situations, >>>>>>>>> they >>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> implement >>>>>>>>>>>>>>>>>>>>>>>>> customized logic using >>>>>>>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned(). >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 < >>>>>>>>> [email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hello chia, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value >>>>>>>>> should >>>>>>>>>>> always >>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right? >>>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive >>>>>>>>> any >>>>>>>>>>>>>>>>>>> field >>>>>>>>>>>>>>>>>>>>>>>>>> absent UnsupportedVersionException >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the >>>>>>> version >>>>>>>>>>> mismatch >>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> during >>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which >>>>>>> occurs >>>>>>>>> inside >>>>>>>>>>>>>>>>>> poll(). >>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the >>>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an >>>>>>> UnsupportedVersionException >>>>>>>>> from >>>>>>>>>>>>>>> poll(). >>>>>>>>>>>>>>>>>>> I’ll >>>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 >>>>>>> 2026年5月17日 >>>>>>>>>>> 下午4:50 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of >>>>>>> this >>>>>>>>>>>>>>> partition >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker >>>>>>> as >>>>>>>>>>>>>>>>>>>>>> broker_current_time >>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker >>>>>>>>> does not >>>>>>>>>>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value >>>>>>>>> should >>>>>>>>>>> always >>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>> greater >>>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right? >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive >>>>>>>>> any >>>>>>>>>>>>>>>>>>> field >>>>>>>>>>>>>>>>>>>>>>>>>> absent UnsupportedVersionException >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException >>>>>>> when >>>>>>>>>>> calling >>>>>>>>>>>>>>>>>>>>>> `poll()`? >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change >>>>>>> based on >>>>>>>>> the >>>>>>>>>>>>>>>>>>> discussion >>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition >>>>>>> reset >>>>>>>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to >>>>>>> all >>>>>>>>>>>>>>>>>>> auto.offset.reset >>>>>>>>>>>>>>>>>>>>>>>>>> values >>>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). >>>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer >>>>>>> resets to >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions >>>>>>>>>>>>>>>>>>>>>>>>>>>> config setting >>>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base >>>>>>>>>>>>>>> auto.offset.reset >>>>>>>>>>>>>>>>>>>>>> policy >>>>>>>>>>>>>>>>>>>>>>>>>> continues >>>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged. >>>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is >>>>>>> represented >>>>>>>>> by a >>>>>>>>>>>>>>>> separate >>>>>>>>>>>>>>>>>>>>>>>>>> internal config >>>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently >>>>>>> fixed >>>>>>>>> to >>>>>>>>>>>>>>>>>> earliest). >>>>>>>>>>>>>>>>>>>>>> This >>>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes >>>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a >>>>>>>>> public >>>>>>>>>>>>>>>>>>> user-facing >>>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 >>>>>>> 2026年5月16日 >>>>>>>>>>> 清晨7:46 >>>>>>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me >>>>>>> is >>>>>>>>> listed >>>>>>>>>>>>>>>> below: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a >>>>>>>>> default >>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both >>>>>>>>> by_duration and >>>>>>>>>>>>>>>>>> latest, >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of >>>>>>>>> auto.offset.reset=earliest. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an >>>>>>>>>>> internal >>>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new >>>>>>>>> .partitions=earliest >>>>>>>>>>>>>>>> already >>>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use >>>>>>> cases >>>>>>>>> of >>>>>>>>>>> other >>>>>>>>>>>>>>>>>>> values >>>>>>>>>>>>>>>>>>>>>>>> in a >>>>>>>>>>>>>>>>>>>>>>>>>> separate KIP. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new >>>>>>> .partitions >>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be >>>>>>> applied >>>>>>>>> to >>>>>>>>>>> all >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>> consistency. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says >>>>>>> "When a >>>>>>>>>>> Kafka >>>>>>>>>>>>>>>>>> topic >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>> expanded >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the >>>>>>> latest >>>>>>>>> auto >>>>>>>>>>>>>>> offset >>>>>>>>>>>>>>>>>>> reset >>>>>>>>>>>>>>>>>>>>>>>>>> policy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to >>>>>>> those >>>>>>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>>> before >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same >>>>>>>>> record >>>>>>>>>>> loss >>>>>>>>>>>>>>>>>> issue >>>>>>>>>>>>>>>>>>>>>> could >>>>>>>>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new >>>>>>>>> .partitions >>>>>>>>>>> will >>>>>>>>>>>>>>>> take >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could >>>>>>> set >>>>>>>>> it >>>>>>>>>>>>>>>>>>> by_duration if >>>>>>>>>>>>>>>>>>>>>>>>>> needed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping >>>>>>> Tsai < >>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing >>>>>>>>> something >>>>>>>>>>>>>>>>>>> important >>>>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try >>>>>>> to >>>>>>>>>>> clarify >>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>> few >>>>>>>>>>>>>>>>>>>>>>>>>> questions: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending >>>>>>> this >>>>>>>>>>> logic to >>>>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>>>>>>> reset >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like >>>>>>> earliest >>>>>>>>> or >>>>>>>>>>>>>>>>>> by_duration >>>>>>>>>>>>>>>>>>>>>>>> don't >>>>>>>>>>>>>>>>>>>>>>>>>> seem >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss >>>>>>> issue >>>>>>>>> when a >>>>>>>>>>>>>>>>>>> partition >>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>> expanded. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to >>>>>>>>> configure for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they >>>>>>> set >>>>>>>>> it to >>>>>>>>>>>>>>>>>> earliest >>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>> latest, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. >>>>>>> For >>>>>>>>>>> example, >>>>>>>>>>>>>>>>>> if a >>>>>>>>>>>>>>>>>>>>>>>>>> consumer is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is >>>>>>>>> created >>>>>>>>>>> during >>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>> downtime, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to >>>>>>> latest >>>>>>>>> when >>>>>>>>>>>>>>>>>> resuming, >>>>>>>>>>>>>>>>>>>>>>>> rather >>>>>>>>>>>>>>>>>>>>>>>>>> than >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the >>>>>>> partition >>>>>>>>> is >>>>>>>>>>>>>>>>>>> technically >>>>>>>>>>>>>>>>>>>>>>>>>> "new" to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing >>>>>>> a >>>>>>>>>>> max.age >>>>>>>>>>>>>>>>>>> threshold. >>>>>>>>>>>>>>>>>>>>>>>> It >>>>>>>>>>>>>>>>>>>>>>>>>> gives >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is >>>>>>>>> genuinely >>>>>>>>>>>>>>>> "hot/new" >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>> what is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a >>>>>>> bit >>>>>>>>>>> weird. It >>>>>>>>>>>>>>>>>> only >>>>>>>>>>>>>>>>>>>>>>>>>> applies when >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it >>>>>>> seems >>>>>>>>> that >>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> motivation >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to >>>>>>> other >>>>>>>>>>> values >>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>>>>>> by_duration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate >>>>>>>>> way to >>>>>>>>>>>>>>>> control >>>>>>>>>>>>>>>>>>>>>> newly >>>>>>>>>>>>>>>>>>>>>>>>>> created >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the >>>>>>> group >>>>>>>>>>> starts. >>>>>>>>>>>>>>>>>>> Have we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like >>>>>>> auto.offset.reset.new >>>>>>>>>>>>>>>> .partitions? >>>>>>>>>>>>>>>>>>> If >>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy >>>>>>>>> defaults >>>>>>>>>>> to >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> policy >>>>>>>>>>>>>>>>>>>>>>>>>> used >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it >>>>>>>>>>> explicitly to >>>>>>>>>>>>>>>>>>>>>> customize >>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 < >>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月1日 >>>>>>> 晚上10:37 >>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. >>>>>>> The >>>>>>>>>>>>>>>>>>> PartitionMetadata >>>>>>>>>>>>>>>>>>>>>>>>>> struct >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), >>>>>>>>> computed >>>>>>>>>>>>>>>>>>> server-side >>>>>>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - >>>>>>> partition_creation_time. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when >>>>>>>>>>>>>>>>>>> MembershipManager >>>>>>>>>>>>>>>>>>>>>>>>>> detects >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the >>>>>>>>> metadata >>>>>>>>>>> for >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> affected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, >>>>>>> even >>>>>>>>> if >>>>>>>>>>> the >>>>>>>>>>>>>>>>>> topic >>>>>>>>>>>>>>>>>>> ID >>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum >>>>>>>>> supported >>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection >>>>>>> time. >>>>>>>>> If the >>>>>>>>>>>>>>>>>>> negotiated >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support >>>>>>>>> PartitionAgeMs >>>>>>>>>>> at >>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling >>>>>>>>> back to >>>>>>>>>>>>>>> latest >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>> risking >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible >>>>>>> signal. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 >>>>>>>>> 2026年4月29日 >>>>>>>>>>>>>>>> 下午4:04 >>>>>>>>>>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the >>>>>>>>> 'age' >>>>>>>>>>>>>>>>>> resolution >>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep >>>>>>> the >>>>>>>>>>> control >>>>>>>>>>>>>>>>>> plane >>>>>>>>>>>>>>>>>>>>>>>> clean. >>>>>>>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is >>>>>>> introducing >>>>>>>>>>>>>>>> inter-broker >>>>>>>>>>>>>>>>>>>>>> clock >>>>>>>>>>>>>>>>>>>>>>>>>> skew. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a >>>>>>> single >>>>>>>>>>> source of >>>>>>>>>>>>>>>>>> truth >>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>> time. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew >>>>>>>>> should be >>>>>>>>>>>>>>>>>>> negligible. >>>>>>>>>>>>>>>>>>>>>>>>>> Given >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be >>>>>>>>> configured in >>>>>>>>>>>>>>>> minutes >>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>> hours, a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between >>>>>>>>> brokers >>>>>>>>>>>>>>> won't >>>>>>>>>>>>>>>>>>> impact >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev < >>>>>>> [email protected]> >>>>>>>>> 於 >>>>>>>>>>>>>>>>>> 2026年4月29日 >>>>>>>>>>>>>>>>>>>>>>>> 下午3:29 >>>>>>>>>>>>>>>>>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the >>>>>>>>> previous >>>>>>>>>>>>>>>>>>> conversation >>>>>>>>>>>>>>>>>>>>>>>> but I >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the >>>>>>> flow >>>>>>>>> with >>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a >>>>>>> new >>>>>>>>>>> boolean; >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> KIP >>>>>>>>>>>>>>>>>>>>>>>>>> states >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when >>>>>>> this >>>>>>>>>>> boolean >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> set. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a >>>>>>> new >>>>>>>>>>>>>>> partition, >>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to >>>>>>> receive >>>>>>>>> the >>>>>>>>>>> ages. >>>>>>>>>>>>>>>> Is >>>>>>>>>>>>>>>>>>> my >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the >>>>>>> flow >>>>>>>>> and >>>>>>>>>>> also >>>>>>>>>>>>>>>>>>> explain >>>>>>>>>>>>>>>>>>>>>>>>>> how it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list >>>>>>> offsets, >>>>>>>>> fetch >>>>>>>>>>>>>>>>>> offsets, >>>>>>>>>>>>>>>>>>>>>>>> etc.). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I >>>>>>>>> wonder >>>>>>>>>>> if >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the >>>>>>> right >>>>>>>>>>> place >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>> given >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. >>>>>>> Alternatively, it >>>>>>>>>>> could >>>>>>>>>>>>>>>>>> simply >>>>>>>>>>>>>>>>>>>>>>>>>> include >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather >>>>>>>>>>> cautious >>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with >>>>>>>>> unrelated >>>>>>>>>>>>>>>> concepts. >>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>> API >>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or >>>>>>> revoking >>>>>>>>>>>>>>>> partitions. >>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>> fact >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the >>>>>>> corresponding >>>>>>>>>>> Streams >>>>>>>>>>>>>>> API >>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>>>>> suggests >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What >>>>>>> would we >>>>>>>>> do >>>>>>>>>>> if we >>>>>>>>>>>>>>>>>>> want to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM >>>>>>> Muralidhar >>>>>>>>>>> Basani >>>>>>>>>>>>>>>> via >>>>>>>>>>>>>>>>>>> dev >>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to >>>>>>> know >>>>>>>>>>> about >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> gap. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for >>>>>>>>>>>>>>>>>>> RequestPartitionAges >>>>>>>>>>>>>>>>>>>>>>>>>> field. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, >>>>>>>>>>> PartitionAges on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a >>>>>>>>>>> recommended >>>>>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on >>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms >>>>>>>>>>>>>>>>>>>>>>>>>> ? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I >>>>>>>>> guess. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes >>>>>>> to >>>>>>>>> Kafka >>>>>>>>>>>>>>>>>> Streams, >>>>>>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config >>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure >>>>>>> it? >>>>>>>>> This >>>>>>>>>>> is >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> most >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily >>>>>>>>>>> mistakenly >>>>>>>>>>>>>>>>>>> configure >>>>>>>>>>>>>>>>>>>>>>>>>> it. Or >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the >>>>>>> consumer >>>>>>>>> falls >>>>>>>>>>>>>>> back >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>> earliest" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed >>>>>>>>> per-partition >>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>> isn't >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like >>>>>>> "consumer >>>>>>>>>>> resolves >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> initial >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if >>>>>>>>>>> earliest was >>>>>>>>>>>>>>>>>>> applied >>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset >>>>>>>>> config is >>>>>>>>>>>>>>>>>>> unchanged. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 < >>>>>>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this >>>>>>>>> change. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> >>>>>>> 於 >>>>>>>>>>>>>>> 2026年4月28日 >>>>>>>>>>>>>>>>>>> 晚上8:03 >>>>>>>>>>>>>>>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the >>>>>>> partition >>>>>>>>>>> creation >>>>>>>>>>>>>>>> time >>>>>>>>>>>>>>>>>>> via >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for >>>>>>> users to >>>>>>>>>>>>>>> diagnose >>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of >>>>>>>>> auto.offset.reset.latest.max.age >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion >>>>>>> on >>>>>>>>>>> KIP-1327 >>>>>>>>>>>>>>>>>>> Prevent >>>>>>>>>>>>>>>>>>>>>> Hot >>>>>>>>>>>>>>>>>>>>>>>>>> Data >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish >>>>>>> newly >>>>>>>>>>> expanded >>>>>>>>>>>>>>>>>> (hot) >>>>>>>>>>>>>>>>>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured >>>>>>> threshold >>>>>>>>>>>>>>>> automatically >>>>>>>>>>>>>>>>>>> fall >>>>>>>>>>>>>>>>>>>>>>>>>> back >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without >>>>>>> forcing a >>>>>>>>>>> full >>>>>>>>>>>>>>>>>>> historical >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >>>> >>> >>
