Hi Feng, Thank you for the thoughtful feedback.
My initial proposal of a single default freshness could cause problems because it doesn't correctly handle the case where a user explicitly selects REFRESH_MODE = FULL while omitting FRESHNESS. As you pointed out, the 3 min default is designed for CONTINUOUS mode and would cause the query to fail validation for FULL mode, as it's not a valid cron interval. Your suggestion to have different defaults for each mode is the correct, robust solution. To address this, I will update the FLIP to propose two new configuration options: 1. materialized-table.default-freshness.continuous: with a default of 3 min. 2. materialized-table.default-freshness.full: with a default of something safe and batch-appropriate. The default value is yet to be determined. The DefaultMaterializedTableEnricher would then use the logicalRefreshMode to determine which default to apply. This would solve the issue you've identified. Does this sound reasonable to you? To help refine this, I have a couple of follow-up questions for you and the community, as your experience here would be very valuable: 1. In your experience, how often do users explicitly define REFRESH_MODE without also defining FRESHNESS? Understanding the commonality of this pattern would help gauge the impact. 2. What would you consider a sensible and safe default freshness for FULL mode? Thanks again for helping to improve the proposal. Best, Ramin On Sat, Oct 11, 2025 at 8:54 AM Feng Jin <[email protected]> wrote: > Hi, Ramin > > Thank you for proposing this FLIP—it will indeed enhance convenience. > > However, regarding Full mode and Continuous mode, should we set different > default values for "freshness"? > > If a user explicitly selects Full mode, should we configure a longer > default freshness period instead? > > > Best, > Feng > > > On Sat, Oct 11, 2025 at 12:02 PM Ron Liu <[email protected]> wrote: > > > Hi, Ramin > > > > Thanks for your response. The updated FLIP looks good to me. > > > > Best, > > Ron > > > > Ramin Gharib <[email protected]> 于2025年10月10日周五 20:39写道： > > > > > Hi Ron, > > > > > > Thank you for the detailed and constructive feedback on the FLIP. These > > are > > > all excellent points. > > > > > > I have updated the FLIP's Confluence page, but I'd like to respond to > > each > > > of your questions here as well. > > > > > > 1. On the Default Freshness Value and Configuration > > > > > > I agree that 3 minutes is a more sensible and safer default than 1 > > minute. > > > To address this, I've updated the FLIP to include a new configuration > > > option, materialized-table.default-freshness, with a default value of 3 > > > min. > > > This provides the best of both worlds: a safer default for everyone and > > the > > > flexibility for administrators to configure it. > > > > > > 2. On the IntervalFreshness API Change (String to int) > > > > > > The FLIP now clarifies that we will use a non-breaking, "deprecate and > > add" > > > strategy. We will keep the existing String getInterval() method but > mark > > it > > > as @Deprecated, and a new, type-safe int getIntervalInt() method will > be > > > added. This ensures existing code continues to compile and function > while > > > encouraging new development to use the safer method. I have updated the > > > FLIP (Section 3.2) to make this explicit. > > > > > > 3. On the Scope and Discovery of MaterializedTableEnricher > > > > > > To keep the initial scope of this FLIP focused and deliverable, I've > > > proposed that the framework will directly use the > > > DefaultMaterializedTableEnricher for now. This provides a complete, > > working > > > feature out of the box. The mechanism for discovering and configuring > > > *custom* enrichers (e.g., via a service loader or environment > > > configuration) is a crucial topic, but I believe it warrants its own > > > dedicated discussion in a follow-up FLIP. This allows us to land this > > > improvement pragmatically. > > > > > > 4. On RefreshContext > > > > > > You're right, the FLIP was missing the concrete design for this. I have > > > updated the FLIP to include the full class definition for > RefreshContext > > in > > > the "Public Interfaces" section (now Section 3.6). It provides the > > enricher > > > with the resolved freshness, logical refresh mode, and the freshness > > > threshold from the configuration. > > > > > > Thanks again for helping to improve the proposal. Please let me know if > > > these updates and the revised FLIP address your questions. > > > > > > Best, > > > > > > Ramin > > > > > > On Thu, Oct 9, 2025 at 5:34 AM Ron Liu <[email protected]> wrote: > > > > > > > Hi, Ramin > > > > > > > > Thanks for initiating this FLIP., This proposal can lower the > threshold > > > of > > > > using Materialized Table, +1. > > > > > > > > I read through the FLIP, and I have the following questions: > > > > > > > > 1. Regarding the default freshness default value, you mentioned that > it > > > is > > > > 1min, how is this value determined? In continuous mode, we will > convert > > > > freshness to checkpoint interval, in our internal best practice, the > > > > checkpoint interval is usually 3min. > > > > In addition, for data lake scenarios such as stream writing paimon, > it > > is > > > > more reasonable to set the checkpoint interval to 3min. In our > internal > > > > best practice, the checkpoint interval is usually 3min. > > > > I think this default value needs to be discussed, or similar to > > > > `materialized-table.refresh-mode.freshness-threshold`, we also > provide > > an > > > > option and provide a default value, which can be configured by the > > user. > > > > > > > > 2. You mentioned that to improve IntervalFreshness, change the > current > > > > string type to int for `interval`? The `getInterval()` public method > > is > > > > already used by Paimon and other external systems, if we change the > > > return > > > > value type, it will cause incompatible behavior. > > > > > > > > 3. About `MaterializedTableEnricher` interface, how to pass the > > > > user-defined implementation to the framework, i.e., how the framework > > > > perceives it? Also, does the scope of this interface support > > > customization > > > > at the table level, or the Catalog level? Or Service level? > > > > > > > > 4. Regarding RefreshContext, can you give a concrete design of the > > > > interface and what information it will provide to > > > > MaterializedTableEnricher? > > > > > > > > > > > > Best, > > > > Ron > > > > > > > > Ramin Gharib <[email protected]> 于2025年10月6日周一 20:19写道： > > > > > > > > > Hi everyone, > > > > > > > > > > I would like to start a discussion on FLIP-551 [1], which proposes > > > making > > > > > the FRESHNESS clause optional for Materialized Tables. > > > > > > > > > > The goal is to simplify the user experience for common streaming > > cases > > > > and > > > > > to enable more powerful, platform-level default logic. > > > > > > > > > > The proposal achieves this by: > > > > > > > > > > 1. > > > > > > > > > > Making the FRESHNESS DDL clause optional, falling back to a > > sensible > > > > > default of 1 minute. > > > > > 2. > > > > > > > > > > Introducing a new, MaterializedTableEnricher interface for > custom > > > > logic > > > > > to resolve the final Freshness and RefreshMode into the > > > > > ResolvedCatalogMaterializedTable when they are omitted. > > > > > > > > > > This change reduces boilerplate for users while providing a clean > > > > extension > > > > > point for advanced use cases. The complete architectural details > and > > > API > > > > > are in the FLIP. > > > > > > > > > > I'm looking forward to hearing your feedback. > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-551%3A+Make+FRESHNESS+Optional+for+Materialized+Tables > > > > > > > > > > Thanks, > > > > > > > > > > Ramin > > > > > > > > > > > > > > >
