Hi, dev, Thank you all for the helpful information and discussion. I have updated FLIP according to the discussion. The main changes are as follows:
<1> Proposed Changes will support extending the SerDes to a local file to Flink's FileSystem and support SQL client gateway mode usage. <2> Proposed Changes will complete the user story of FLIP-190 to add syntax support EXPLAIN PLAN 'plan.json' to let users get insight from the compiled JSON file. <3> Add a char to End-to-End Examples to explain the scope of use of this feature, which will not affect other users who do not have this requirement. <4> Point out the current state TTL unalignment issue and the probability of magnifying this unalignment by enabling fine-grained state TTL. This will also be explained in the document to inform users. If you have any questions, please let me know. Best, Jane On Sun, Mar 26, 2023 at 12:31 AM Jane Chan <qingyue....@gmail.com> wrote: > Hi Benchao, > > Thank you for sharing the discussion thread with us; it's very > informational and helpful. Let's set aside the semantics for a moment, and > let me respond to the rest of your points. > > > SQL + hints may be the easiest way for users to write and maintain their > SQL jobs. Of course, hints are not perfect due to its propagation process > is not very clear for end users, IMO, this is not a problem since it's > similar to SQL itself, users are usually not that easy to understand the > physical execution graph compiled from the SQL too, especially new comers. > > I think it is only partially correct to say that users do not understand > or need to care about the SQL compilation. > > In fact, it is the user who does not need to use hints or other advanced > configurations like optimizing the state usage, that does not need to care > about it. > > However, once users need to use hints, it means that they are confident > that they're smarter than the optimizer, know their data distribution > better, and are able to use hints to guide the optimizer to obtain a better > plan. I don't think these users will be novice users. In this case, would > they prefer to use a hint with an unclear propagation mechanism to > configure fine-grained TTL with a guess, or would they prefer to have a > complete file to present which operators are stateful and then configure > them? > > If a novice user doesn't understand anything, how could he have a need to > tune fine-grained TTL? He may not even know what hint/TTL is. This seems to > be a paradox. If by any chance, he is required to do such a complex task, > presenting him with a structured JSON file and listing clearly which > operator has a TTL JSON key (which implies "what you need to do is just > fill in the blank aka. modify the corresponding JSON value") is much better > than making himself lost in the doc pages of join/agg/row_number/etc. and > trying to understand which operation may contain a stateful computation. > > > I think Shuo's idea is very good that hints and json plan may not > contradict each other. Finally we may have multi level configurations, > e.g., hints -> json plan -> job level configurations -> cluster level > configurations, top level configuration overrides low level's. > > Before rushing to discuss whether to introduce multiple levels of state > TTL configuration, I think we should first discuss how to clearly map query > blocks to the underlying stateful operators and let users understand. Maybe > in another FLIP. > > Best, > Jane > > On Sat, Mar 25, 2023 at 4:12 PM Benchao Li <libenc...@apache.org> wrote: > >> Thanks Jane for starting this discussion. Finally we are considering >> fine-grained configurations for SQL jobs, this is very exciting! >> >> I see a lot of opinions that "SQL hints should not affect the results", >> and >> this indeed has been discussed extensively while we introduced >> FLIP-113[1][2]. After three years, I still think that in streaming SQL, >> there are a lot of new things that are different from traditional bounded >> SQL, and hint may be one of them. >> >> Most people have raised a valid concern about the ease of use. SQL + hints >> may be the easiest way for users to write and maintain their SQL jobs. Of >> course, hints are not perfect due to its propagation process is not very >> clear for end users, IMO, this is not a problem since it's similar to SQL >> itself, users are usually not that easy to understand the physical >> execution graph compiled from the SQL too, especially new comers. >> >> I think Shuo's idea is very good that hints and json plan may not >> contradict each other. Finally we may have multi level configurations, >> e.g., hints -> json plan -> job level configurations -> cluster level >> configurations, top level configuration overrides low level's. >> >> [1] >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL >> [2] https://lists.apache.org/thread/78s8bqdql65o61742yb2rtjff66m166r >> >> Jing Ge <j...@ververica.com.invalid> 于2023年3月25日周六 07:26写道： >> >> > Thanks Jane for driving this FLIP. >> > >> > The FLIP is quite interesting. Since the execution plan has finer >> > granularity than the plain SQL script, Hints at SQL level might not be >> able >> > to touch specific operators, which turns out that the idea of leveraging >> > the compiled execution plan is brilliant. >> > >> > However, there are some concerns that might need to be considered. >> > >> > - One thing I didn't fully understand. I might be wrong. Could those ttl >> > configs be survived when SQL jobs are restarted? Does that mean that, >> once >> > I modified the compiled sql plan, the json file will become the sql >> job? I >> > have to always call the EXECUTE PLAN every time when the job needs to be >> > restarted? In case that the original SQL script has been changed, we >> need >> > to compile a version2 sql plan and copy the ttl configs from version1 >> sql >> > plan to version2 and drop version1. This means we have to keep the >> compiled >> > json file and create a link with the original SQL script. I am not sure >> if >> > I understood it correctly, it seems like a lot of maintenance effort. >> > - If I am not mistaken, the compiled sql plan introduced by FLIP-190 is >> > only used for SQL job migration/update. Common stages that Flink uses to >> > produce the execution plan from SQL does not contain the compiling step. >> > This makes one tool do two different jobs[1], upgrade + ttl tuning. >> > and tighten the dependency on compiling sql plans. Flink SQL users have >> to >> > deal with a compiled sql plan for performance optimization that is not >> > designed for it. >> > - The regular working process for Flink SQL users is changed, from only >> > dealing with SQL like scripts to moving between SQL like scripts and >> file >> > modifications back and forth. This is a big change for user behaviours. >> One >> > option could be that we upgrade/extend the COMPILE PLAN to allow users >> > update ttl for operators at the script level. But I am not sure if it is >> > possible to point out specific operators at this level. Another option >> is >> > to print out the result of COMPILE PLAN and enable EXECUTE PLAN 'json >> plan >> > as string'. Third option is to leverage a data platform to virtualize >> the >> > compiled sql plan and provide related interactions for updating ttl and >> > submit(execute) the modified compiled sql plan. >> > >> > On the other side, there is one additional benefit with this proposal: >> we >> > could fine tune SQL jobs while we migrate/upgrade them. That is nice! >> > >> > Best regards, >> > Jing >> > >> > [1] https://en.wikipedia.org/wiki/Single-responsibility_principle >> > >> > On Fri, Mar 24, 2023 at 4:02 PM Leonard Xu <xbjt...@gmail.com> wrote: >> > >> > > Thanks Jane for the proposal. >> > > >> > > TTL of state is an execution phase configuration, serialized json >> graph >> > > file is the graph for execution phase, supporting the operator level >> > state >> > > TTL in the execution json file makes sense to me. >> > > >> > > From the user's perspective, I have two concerns: >> > > 1. By modifying the execution graph node configuration, this raises >> the >> > > cost for users to understand, especially for SQL users. >> > > 2. Submitting a SQL job through `exec plan json file` is not so >> intuitive >> > > as users cannot see the SQL detail of the job >> > > >> > > Best, >> > > Leonard >> > > >> > > On Fri, Mar 24, 2023 at 5:07 PM Shengkai Fang <fskm...@gmail.com> >> wrote: >> > > >> > > > Hi, Jane. >> > > > >> > > > Thanks for driving this FLIP and this feature are very useful to >> many >> > > > users. But I have two problems about the FLIP: >> > > > >> > > > 1. How the Gateway users use this feature? As far as I know, the >> > EXEUCTE >> > > > PLAN only supports local file right now. Is it possible to extend >> this >> > > > syntax to allow for reading plan files from remote file systems? >> > > > >> > > > 2. I would like to inquire if there are any limitations on this >> > feature? >> > > I >> > > > have encountered several instances where the data did not expire in >> the >> > > > upstream operator, but it expired in the downstream operator, >> resulting >> > > in >> > > > abnormal calculation results or direct exceptions thrown by the >> > operator >> > > > (e.g. rank operator). Can we limit that the expiration time of >> > downstream >> > > > operator data should be greater than or equal to the expiration >> time of >> > > > upstream operator data? >> > > > >> > > > Best, >> > > > Shengkai >> > > > >> > > > Yun Tang <myas...@live.com> 于2023年3月24日周五 14:50写道： >> > > > >> > > > > Hi, >> > > > > >> > > > > From my point of view, I am a bit against using SQL hint to set >> state >> > > TTL >> > > > > as FlinkSQL could be translated to several stateful operators. If >> we >> > > want >> > > > > to let different state could have different TTL configs within one >> > > > > operator, the SQL hint solution could not work. A better way is to >> > > allow >> > > > a >> > > > > graphical IDE to display the stateful operators and let users >> > configure >> > > > > them. And the IDE submits the json plan to Flink to run jobs. >> > > > > >> > > > > For the details of the structure of ExecNodes, since the state >> name >> > is >> > > > > unique in the underlying state layer, shall we introduce the >> "index" >> > > tag >> > > > to >> > > > > identify the state config? >> > > > > What will happen with the conditions below: >> > > > > 1st run: >> > > > > { >> > > > > "index": 0, >> > > > > "ttl": "259200000 ms", >> > > > > "name": "join-lef-state" >> > > > > }, >> > > > > { >> > > > > "index": 1, >> > > > > "ttl": "86400000 ms", >> > > > > "name": "join-right-state" >> > > > > } >> > > > > >> > > > > 2nd run: >> > > > > { >> > > > > "index": 0, >> > > > > "ttl": "86400000 ms", >> > > > > "name": "join-right-state" >> > > > > }, >> > > > > { >> > > > > "index": 1, >> > > > > "ttl": "259200000 ms", >> > > > > "name": "join-lef-state" >> > > > > } >> > > > > >> > > > > Best >> > > > > Yun Tang >> > > > > ________________________________ >> > > > > From: Jane Chan <qingyue....@gmail.com> >> > > > > Sent: Friday, March 24, 2023 11:57 >> > > > > To: dev@flink.apache.org <dev@flink.apache.org> >> > > > > Subject: Re: [DISCUSS] FLIP-292: Support configuring state TTL at >> > > > operator >> > > > > level for Table API & SQL programs >> > > > > >> > > > > Hi Shammon and Shuo, >> > > > > >> > > > > Thanks for your valuable comments! >> > > > > >> > > > > Some thoughts: >> > > > > >> > > > > @Shuo >> > > > > > I think it's more properly to say that hint does not affect the >> > > > > equivalenceof execution plans (hash agg vs sort agg), not the >> > > equivalence >> > > > > of execution >> > > > > results, e.g., users can set 'scan.startup.mode' for kafka >> connector >> > by >> > > > > dynamic table option, which >> > > > > also "intervene in the calculation of data results". >> > > > > >> > > > > IMO, the statement that "hint should not interfere with the >> > calculation >> > > > > results", means it should not interfere with internal >> computation. On >> > > the >> > > > > other hand, 'scan.startup.mode' interferes with the ingestion of >> the >> > > > data. >> > > > > I think these two concepts are different, but of course, this is >> just >> > > my >> > > > > opinion and welcome other views. >> > > > > >> > > > > > I think the final shape of state ttl configuring may like the >> that, >> > > > > userscan define operator state ttl using SQL HINT (assumption...), >> > but >> > > it >> > > > > may >> > > > > affects more than one stateful operators inside the same query >> block, >> > > > then >> > > > > users can further configure a specific one by modifying the >> compiled >> > > json >> > > > > plan... >> > > > > >> > > > > Setting aside the issue of semantics, setting TTL from a higher >> level >> > > > seems >> > > > > to be attractive. This means that users only need to configure >> > > > > 'table.exec.state.ttl' through the existing hint syntax to achieve >> > the >> > > > > effect. Everything is a familiar formula. But is it really the >> case? >> > > > Hints >> > > > > apply to a very broad range. Let me give an example. >> > > > > >> > > > > Suppose a user wants to set different TTLs for the two streams in >> a >> > > > stream >> > > > > join query. Where should the hints be written? >> > > > > >> > > > > -- the original query before configuring state TTL >> > > > > create temporary view view1 as select .... from my_table_1; >> > > > > create temporary view view2 as select .... from my_table_2; >> > > > > create temporary view joined_view as >> > > > > select view1.*, view2.* from my_view_1 a join my_view_2 b on >> > > a.join_key = >> > > > > b.join_key; >> > > > > >> > > > > Option 1: declaring hints at the very beginning of the table scan >> > > > > >> > > > > -- should he or she write hints when declaring the first temporary >> > > view? >> > > > > create temporary view view1 as select .... from my_table_1 >> > > > > /*+(OPTIONS('table.exec.state.ttl' >> > > > > = 'foo'))*/; >> > > > > create temporary view view2 as select .... from my_table_2 >> > > > > /*+(OPTIONS('table.exec.state.ttl' >> > > > > = 'bar'))*/; >> > > > > create temporary view joined_view as >> > > > > select view1.*, view2.* from my_view_1 a join my_view_2 b on >> > > a.join_key = >> > > > > b.join_key; >> > > > > >> > > > > Option 2: declaring hints when performing the join >> > > > > >> > > > > -- or should he or she write hints when declaring the join >> temporary >> > > > view? >> > > > > create temporary view view1 as select .... from my_table_1; >> > > > > create temporary view view2 as select .... from my_table_2; >> > > > > create temporary view joined_view as >> > > > > select view1.*, view2.* from my_view_1 >> > > > /*+(OPTIONS('table.exec.state.ttl' = >> > > > > 'foo'))*/ a join my_view_2 /*+(OPTIONS('table.exec.state.ttl' = >> > > > 'bar'))*/ b >> > > > > on a.join_key = b.join_key; >> > > > > >> > > > > From the user's point of view, does he or she needs to care about >> the >> > > > > difference between these two kinds of style? Users might think the >> > two >> > > > may >> > > > > be equivalent; but in reality, as developers, how do we define the >> > > range >> > > > in >> > > > > which hint starts and ends to take effect? >> > > > > >> > > > > Consider the following two assumptions >> > > > > >> > > > > 1. Assuming the hint takes effect from the moment it is declared >> and >> > > > > applies to any subsequent stateful operators until it is >> overridden >> > by >> > > a >> > > > > new hint. >> > > > > If this is the assumption, it's clear that Option 1 and Option 2 >> are >> > > > > different because a ChangelogNormalize node can appear between >> scan >> > and >> > > > > join. Meanwhile, which stream's TTL to apply to the following >> query >> > > after >> > > > > the stream join? It is unclear if the user does not explicitly set >> > it. >> > > > > Should the engine make a random decision? >> > > > > >> > > > > 2. Assuming that the scope of the hint only applies to the current >> > > query >> > > > > block and does not extend to the next operator. >> > > > > In this case, the first way of setting the hint will not work >> because >> > > it >> > > > > cannot be brought to the join operator. Users must choose the >> second >> > > way >> > > > to >> > > > > configure. Are users willing to remember this strange constraint >> on >> > SQL >> > > > > writing style? Does this indicate a new learning cost? >> > > > > >> > > > > The example above is used to illustrate that while this approach >> may >> > > seem >> > > > > simple and direct, it actually has many limitations and may >> produce >> > > > > unexpected behavior. Will users still find it attractive? IMO >> *hints >> > > only >> > > > > work for a very limited situation where the query is very simple, >> and >> > > its >> > > > > scope is more coarse and not operator-level*. Maybe it deserves >> > another >> > > > > FLIP to discuss whether we need a multiple-level state TTL >> > > configuration >> > > > > mechanism and how to properly implement it. >> > > > > >> > > > > @Shammon >> > > > > > Generally, Flink jobs support two types >> > > > > of submission: SQL and jar. If users want to use `TTL on Operator` >> > for >> > > > SQL >> > > > > jobs, they need to edit the json file which is not supported by >> > general >> > > > job >> > > > > submission systems such as flink sql-client, apache kyuubi, apache >> > > > > streampark and .etc. Users need to download the file and edit it >> > > > manually, >> > > > > but they may not have the permissions to the storage system such >> as >> > > HDFS >> > > > in >> > > > > a real production environment. From this perspective, I think it >> is >> > > > > necessary to provide a way similar to >> > > > > hits that users can configure the `TTL on Operator` in their sqls >> > which >> > > > > help users to use it conveniently. >> > > > > >> > > > > IIUC, SQL client supports the statement "EXECUTE PLAN >> > > > > 'file:/foo/bar/example.json'". While I think there is not much >> > evidence >> > > > to >> > > > > say we should choose to use hints, just because users cannot touch >> > > their >> > > > > development environment. As a reply to @Shuo, the TTL set through >> > hint >> > > > way >> > > > > is not at the operator level. And whether it is really >> "convenient" >> > > needs >> > > > > more discussion. >> > > > > >> > > > > > I agree with @Shuo's idea that for complex cases, users can >> combine >> > > > hits >> > > > > and `json plan` to configure `TTL on Operator` better. >> > > > > >> > > > > Suppose users can configure TTL through >> > > > > <1> SET 'table.exec.state.ttl' = 'foo'; >> > > > > <2> Modify the compiled JSON plan; >> > > > > <3> Use hints (personally I'm strongly against this way, but let's >> > take >> > > > it >> > > > > into consideration). >> > > > > IMO if the user can configure the same parameter in so many ways, >> > then >> > > > the >> > > > > complex case only makes things worse. Who has higher priority and >> who >> > > > > overrides who? >> > > > > >> > > > > Best, >> > > > > Jane >> > > > > >> > > > > >> > > > > On Fri, Mar 24, 2023 at 11:00 AM Shammon FY <zjur...@gmail.com> >> > wrote: >> > > > > >> > > > > > Hi jane >> > > > > > >> > > > > > Thanks for initializing this discussion. Configure TTL per >> operator >> > > can >> > > > > > help users manage state more effectively. >> > > > > > >> > > > > > I think the `compiled json plan` proposal may need to consider >> the >> > > > impact >> > > > > > on the user's submission workflow. Generally, Flink jobs support >> > two >> > > > > types >> > > > > > of submission: SQL and jar. If users want to use `TTL on >> Operator` >> > > for >> > > > > SQL >> > > > > > jobs, they need to edit the json file which is not supported by >> > > general >> > > > > job >> > > > > > submission systems such as flink sql-client, apache kyuubi, >> apache >> > > > > > streampark and .etc. Users need to download the file and edit it >> > > > > manually, >> > > > > > but they may not have the permissions to the storage system >> such as >> > > > HDFS >> > > > > in >> > > > > > a real production environment. >> > > > > > >> > > > > > From this perspective, I think it is necessary to provide a way >> > > similar >> > > > > to >> > > > > > hits that users can configure the `TTL on Operator` in their >> sqls >> > > which >> > > > > > help users to use it conveniently. At the same time, I agree >> with >> > > > @Shuo's >> > > > > > idea that for complex cases, users can combine hits and `json >> plan` >> > > to >> > > > > > configure `TTL on Operator` better. What do you think? Thanks >> > > > > > >> > > > > > >> > > > > > Best, >> > > > > > Shammon FY >> > > > > > >> > > > > > >> > > > > > On Thu, Mar 23, 2023 at 9:58 PM Shuo Cheng <njucs...@gmail.com> >> > > wrote: >> > > > > > >> > > > > > > Correction: “users can set 'scan.startup.mode' for kafka >> > connector” >> > > > -> >> > > > > > > “users >> > > > > > > can set 'scan.startup.mode' for kafka connector by dynamic >> table >> > > > > option” >> > > > > > > >> > > > > > > Shuo Cheng <njucs...@gmail.com>于2023年3月23日 周四21:50写道： >> > > > > > > >> > > > > > > > Hi Jane, >> > > > > > > > Thanks for driving this, operator level state ttl is >> > absolutely a >> > > > > > desired >> > > > > > > > feature. I would share my opinion as following: >> > > > > > > > >> > > > > > > > If the scope of this proposal is limited as an enhancement >> for >> > > > > compiled >> > > > > > > > json plan, it makes sense. I think it does not conflict with >> > > > > > configuring >> > > > > > > > state ttl >> > > > > > > > in other ways, e.g., SQL HINT or something else, because >> they >> > > just >> > > > > work >> > > > > > > in >> > > > > > > > different level, SQL Hint works in the exact entrance of SQL >> > API, >> > > > > while >> > > > > > > > compiled json plan is the intermediate results for SQL. >> > > > > > > > I think the final shape of state ttl configuring may like >> the >> > > that, >> > > > > > users >> > > > > > > > can define operator state ttl using SQL HINT >> (assumption...), >> > but >> > > > it >> > > > > > may >> > > > > > > > affects more than one stateful operators inside the same >> query >> > > > block, >> > > > > > > then >> > > > > > > > users can further configure a specific one by modifying the >> > > > compiled >> > > > > > json >> > > > > > > > plan... >> > > > > > > > >> > > > > > > > In a word, this proposal is in good shape as an enhancement >> for >> > > > > > compiled >> > > > > > > > json plan, and it's orthogonal with other ways like SQL Hint >> > > which >> > > > > > works >> > > > > > > in >> > > > > > > > a higher level. >> > > > > > > > >> > > > > > > > >> > > > > > > > Nips: >> > > > > > > > >> > > > > > > > > "From the SQL semantic perspective, hints cannot >> intervene in >> > > the >> > > > > > > > calculation of data results." >> > > > > > > > I think it's more properly to say that hint does not affect >> the >> > > > > > > > equivalence of execution plans (hash agg vs sort agg), not >> the >> > > > > > > equivalence >> > > > > > > > of execution results, e.g., users can set >> 'scan.startup.mode' >> > for >> > > > > kafka >> > > > > > > > connector, which also "intervene in the calculation of data >> > > > results". >> > > > > > > > >> > > > > > > > Sincerely, >> > > > > > > > Shuo >> > > > > > > > >> > > > > > > > On Tue, Mar 21, 2023 at 7:52 PM Jane Chan < >> > qingyue....@gmail.com >> > > > >> > > > > > wrote: >> > > > > > > > >> > > > > > > >> Hi devs, >> > > > > > > >> >> > > > > > > >> I'd like to start a discussion on FLIP-292: Support >> > configuring >> > > > > state >> > > > > > > TTL >> > > > > > > >> at operator level for Table API & SQL programs [1]. >> > > > > > > >> >> > > > > > > >> Currently, we only support job-level state TTL >> configuration >> > via >> > > > > > > >> 'table.exec.state.ttl'. However, users may expect a >> > fine-grained >> > > > > state >> > > > > > > TTL >> > > > > > > >> control to optimize state usage. Hence we propose to >> > > > > > > serialize/deserialize >> > > > > > > >> the state TTL as metadata of the operator's state to/from >> the >> > > > > compiled >> > > > > > > >> JSON >> > > > > > > >> plan, to achieve the goal that specifying different state >> TTL >> > > when >> > > > > > > >> transforming the exec node to stateful operators. >> > > > > > > >> >> > > > > > > >> Look forward to your opinions! >> > > > > > > >> >> > > > > > > >> [1] >> > > > > > > >> >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240883951 >> > > > > > > >> >> > > > > > > >> Best Regards, >> > > > > > > >> Jane Chan >> > > > > > > >> >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >> >> -- >> >> Best, >> Benchao Li >> >
