Hi Jane,
The fine-grained TTL management is extremely useful for performance
tuning, so +1 for the idea. I have a minor suggestion: would it be
possible to provide a simple hint that allows the omission of the key?
For example, something like "SELECT /+ STATE_TTL('1h')/", which would
specify the TTL for all states in the 'SELECT' clause.
And I also share the same concern as Feng. I am wondering if we could
show the state ttl for operators in Flink UI.
Best,
Zakelly
On Wed, Oct 11, 2023 at 1:27 PM Feng Jin <jinfeng1...@gmail.com> wrote:
>
> Hi Jane,
>
> Thank you for providing this explanation.
>
> Another small question, since there is no exception thrown when the STATE
> hint is set incorrectly,
> should we somehow show that the TTL setting has taken effect?
> For instance, exhibiting the TTL option within the operator's description?
>
> Best,
> Feng
>
> On Tue, Oct 10, 2023 at 7:51 PM Xuyang <xyzhong...@163.com> wrote:
>
> > Hi, Jane.
> >
> >
> > I think this syntax will be easier for users to set operator ttl. So big
> > +1. I left some minor comments here.
> >
> >
> > I notice that using STATE_TTL hints wrongly will not throw any exceptions.
> > But it seems that in the current join hint scenario,
> > if user uses an unknown table name as the chosen side, a validation
> > exception will be thrown.
> > Maybe we should distinguish which exceptions need to be thrown explicitly.
> >
> >
> >
> >
> > --
> >
> > Best！
> > Xuyang
> >
> >
> >
> >
> >
> > At 2023-10-10 18:23:55, "Jane Chan" <qingyue....@gmail.com> wrote:
> > >Hi Feng,
> > >
> > >Thank you for your valuable comments. The reason for not including the
> > >scenarios above is as follows:
> > >
> > >For <1>, the automatically inferred stateful operators are not easily
> > >expressible in SQL. This issue was discussed in FLIP-292, and besides
> > >ChangelogNormalize, SinkUpsertMateralizer also faces the same problem.
> > >
> > >For <2> and <3>, the challenge lies in internal implementation. During the
> > >default_rewrite phase, the row_number expression in LogicalProject is
> > >transformed into LogicalWindow by Calcite's
> > >CoreRules#PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW. However, CalcRelSplitter
> > >does not pass the hints as an input argument when creating LogicalWindow,
> > >resulting in the loss of the hint at this step. To support this, we may
> > >need to rewrite some optimization rules in Calcite, which could be a
> > >follow-up work if required.
> > >
> > >Best,
> > >Jane
> > >
> > >On Tue, Oct 10, 2023 at 1:40 AM Feng Jin <jinfeng1...@gmail.com> wrote:
> > >
> > >> Hi Jane,
> > >>
> > >> Thank you for proposing this FLIP.
> > >>
> > >> I believe that this FLIP will greatly enhance the flexibility of setting
> > >> state, and by setting different operators' TTL, it can also increase job
> > >> stability, especially in regular join scenarios.
> > >> The parameter design is very concise, big +1 for this, and it is also
> > >> relatively easy to use for users.
> > >>
> > >>
> > >> I have a small question: in the FLIP, it only mentions join and group.
> > >> Should we also consider other scenarios?
> > >>
> > >> 1. the auto generated deduplicate operator[1].
> > >> 2. the deduplicate query[2].
> > >> 3. the topN query[3].
> > >>
> > >> [1]
> > >>
> > >>
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-source-cdc-events-duplicate
> > >> [2]
> > >>
> > >>
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/deduplication/
> > >> [3]
> > >>
> > >>
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/topn/
> > >>
> > >>
> > >> Best,
> > >> Feng
> > >>
> > >> On Sun, Oct 8, 2023 at 5:53 PM Jane Chan <qingyue....@gmail.com> wrote:
> > >>
> > >> > Hi devs,
> > >> >
> > >> > I'd like to initiate a discussion on FLIP-373: Support Configuring
> > >> > Different State TTLs using SQL Hint [1]. This proposal is on top of
> > the
> > >> > FLIP-292 [2] to address typical scenarios with unambiguous semantics
> > and
> > >> > hint propagation.
> > >> >
> > >> > I'm looking forward to your opinions!
> > >> >
> > >> >
> > >> > [1]
> > >> >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint
> > >> > [2]
> > >> >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration
> > >> >
> > >> > Best,
> > >> > Jane
> > >> >
> > >>
> >
