Hi, Ramin. Sorry for this late reply. After reading this FLIP, I have a few
questions I’d like to clarify.
1. Regarding STATE_RETENTION: Besides ALL and NONE, I understand that PTF_ONLY
allows users to selectively retain state for specific operators. If the query
is modified and successfully restored, should we be concerned about potential
inconsistencies between the new query’s data and the old data stored in the
retained state? For example, could certain operators produce unexpected or
anomalous results due to such mismatches?
2. On the scope of PTF_ONLY: The FLIP describes it as allowing users to
"specify the names or IDs of specific Process Table Functions (PTFs) or
operators." Are there plans to introduce more granular options in the
future—such as JOIN_ONLY, AGG_ONLY, or JOIN_AND_AGG_ONLY? For advanced users,
would it be more flexible and generic to allow specifying arbitrary operators
directly (e.g., via ExecNodeMetadata#name)?
3. Interaction between STATE_RETENTION and START_MODE: It seems to me that
START_MODE is only meaningful and valid when STATE_RETENTION is set to NONE.
For example, if a user sets STATE_RETENTION = ALL together with START_MODE =
FROM_BEGINNING, the resulting behavior appears counterintuitive—should this
combination even be allowed?
4. Behavior in batch mode: What exactly does STATE_RETENTION do in batch
execution? Is it simply ignored? Ideally, I’d prefer it not to throw an error.
One of MT’s core design goals is to abstract away whether the underlying engine
runs in streaming or batch mode from the user’s perspective. If we require
different handling based on execution mode at the MT layer, it would undermine
this abstraction and force us to introduce additional stream/batch distinctions
in MT’s high-level API design.
Looking forward to your response.
--
Best！
Xuyang
At 2026-02-02 18:51:50, "Ramin Gharib" <[email protected]> wrote:
>Hi everyone,
>
>Thank you to everyone who has participated in the discussion for FLIP-557
>and provided valuable feedback.
>
>I have started the formal [VOTE] thread [1] for this proposal.
>
>Best regards,
>
>Ramin
>
>[1] https://lists.apache.org/thread/c0kqqn6pb60ow4l54sb9cx9z360p7dg5
