Hi Jing, Maybe I was not clear enough, sorry. However the main reason for this item about Calcite rules is not abandoning Scala. The main reason are changes in Calcite itself where there was introduced code generator framework (immutables) to generate config java classes for rules and old api (which is used in Scala Calcirte rules) for that is marked as deprecated. Since Immutables implies code generation while java compilation it seems impossible to use for rules in Scala code.
On Mon, Jul 3, 2023 at 10:44 AM Jing Ge <j...@ververica.com.invalid> wrote: > Hi, > > Speaking of "Move Calcite rules from Scala to Java", I was wondering if > this thread is the right place to talk about it. Afaik, the Flink community > has decided to abandon Scala. That is the reason, I guess, we want to move > those Calcite rules from Scala to Java. On the other side, new Scala code > will be added while developing new features[1]. Do we have any thoughts > wrt the Scala code strategy? > > Best regards, > Jing > > > > [1] https://lists.apache.org/thread/tnygl4n3q1fx75cl2vclc78j8mrxmz6y > > On Mon, Jul 3, 2023 at 10:31 AM Xintong Song <tonysong...@gmail.com> > wrote: > > > Thanks all for the discussion. > > > > > > IIUC, we need to make the following changes. Please correct me if I get > it > > wrong. > > > > > > 1. Disaggregated State Management - Clarify that only the public API > > related part is must-have for 2.0. > > > > 2. Java version support - Split it into 3 items: a) make java 17 the > > default (must-have), b) drop java 8 (must-have), and c) drop java 11 > > (nice-to-have) > > > > 3. Add MetricGroup#getLogicalScope - Should be promoted to must-have > > > > 4. ProcessFunction API - Should be downgrade to nice-to-have > > > > 5. Configuration - Add an item "revisit all config option types and > default > > values", which IIUC should also be a must-have > > > > > > There seems to be no changes needed for "Move Calcite rules from Scala to > > Java" as it's already nice-to-have. > > > > > > If there's no objections, I'll update the wiki page accordingly, and > start > > a VOTE in the next couple of days. > > > > > > Best, > > > > Xintong > > > > > > > > On Fri, Jun 30, 2023 at 12:53 AM Teoh, Hong <lian...@amazon.co.uk.invalid > > > > wrote: > > > > > Thanks Xintong for driving the effort. > > > > > > I’d add a +1 to reworking configs, as suggested by @Jark and @Chesnay, > > > especially the types. We have various configs that encode Time / > > MemorySize > > > that are Long instead! > > > > > > Regards, > > > Hong > > > > > > > > > > > > > On 29 Jun 2023, at 16:19, Yuan Mei <yuanmei.w...@gmail.com> wrote: > > > > > > > > CAUTION: This email originated from outside of the organization. Do > not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > > > > > Thanks for driving this effort, Xintong! > > > > > > > > To Chesnay > > > >> I'm curious as to why the "Disaggregated State Management" item is > > > >> marked as a must-have; will it require changes that break something? > > > >> What prevents it from being added in 2.1? > > > > > > > > As to "Disaggregated State Management". > > > > > > > > We plan to provide a new type of state backend to support DFS as > > primary > > > > storage. > > > > To achieve this, we at least need to include two parts of amends (not > > > > entirely sure yet, since we are still in the designing and prototype > > > phase) > > > > > > > > 1. Statebackend Change > > > > 2. State Access Change > > > > > > > > Not all of the interfaces related are `@Internal`. Some of the > > interfaces > > > > like `StateBackend` is `@PublicEvolving` > > > > So, you are right in the sense that "Disaggregated State Management" > > > itself > > > > probably does not need to be a "Must Have" > > > > > > > > But I was hoping changes that related to public APIs can be finalized > > and > > > > merged in Flink 2.0 (I will fix the wiki accordingly). > > > > > > > > I also agree with Jark that 2.0 is a good chance to rework the > default > > > > value of configurations. > > > > > > > > Best > > > > Yuan > > > > > > > > > > > > On Thu, Jun 29, 2023 at 8:43 PM Chesnay Schepler <ches...@apache.org > > > > > wrote: > > > > > > > >> Something else configuration-related is that there are a bunch of > > > >> options where the type isn't quite correct (e.g., a String where it > > > >> could be an enum, a string where it should be an int or something). > > > >> Could do a pass over those as well. > > > >> > > > >> On 29/06/2023 13:50, Jark Wu wrote: > > > >>> Hi, > > > >>> > > > >>> I think one more thing we need to consider to do in 2.0 is changing > > the > > > >>> default value of configuration to improve out-of-box user > experience. > > > >>> > > > >>> Currently, in order to run a Flink job, users may need to set > > > >>> a bunch of configurations, such as minibatch, checkpoint interval, > > > >>> exactly-once, > > > >>> incremental-checkpoint, etc. It's very verbose and hard to use for > > > >>> beginners. > > > >>> Most of them can have a universally applicable value. Because > > changing > > > >> the > > > >>> default value is a breaking change. I think It's worth considering > > > >> changing > > > >>> them in 2.0. > > > >>> > > > >>> What do you think? > > > >>> > > > >>> Best, > > > >>> Jark > > > >>> > > > >>> > > > >>> On Wed, 28 Jun 2023 at 14:10, Sergey Nuyanzin <snuyan...@gmail.com > > > > > >> wrote: > > > >>> > > > >>>> Hi Chesnay > > > >>>> > > > >>>>> "Move Calcite rules from Scala to Java": I would hope that this > > would > > > >> be > > > >>>>> an entirely internal change, and could thus be an incremental > > process > > > >>>>> independent of major releases. > > > >>>>> What is the actual scale of this item; how much are we actually > > > >>>> re-writing? > > > >>>> > > > >>>> Thanks for asking > > > >>>> yes, you're right, that should be internal change. > > > >>>> Yeah I was also thinking about incremental change (rule by rule or > > > >>>> reasonable small group of rules). > > > >>>> And yes, this could be an independent (on major release) activity > > > >>>> > > > >>>> The problem is actually for children of RelOptRule. > > > >>>> Currently I see 60+ such rules (in Scala) using the mentioned > > > deprecated > > > >>>> api. > > > >>>> There are also children of ConverterRule (50+) which do not have > > such > > > >>>> issues. > > > >>>> Maybe it could be considered as the next step to have all the > rules > > in > > > >>>> Java. > > > >>>> > > > >>>> On Tue, Jun 27, 2023 at 1:34 PM Xintong Song < > tonysong...@gmail.com > > > > > > >>>> wrote: > > > >>>> > > > >>>>> Hi Alex & Gyula, > > > >>>>> > > > >>>>> By compatibility discussion do you mean the "[DISCUSS] FLIP-321: > > > >>>> Introduce > > > >>>>>> an API deprecation process" thread [1]? > > > >>>>>> > > > >>>>> Yes, I meant the FLIP-321 discussion. I just noticed I pasted the > > > wrong > > > >>>> url > > > >>>>> in my previous email. Sorry for the mistake. > > > >>>>> > > > >>>>> I am also curious to know if the rationale behind this new API > has > > > been > > > >>>>>> previously discussed on the mailing list. Do we have a list of > > > >>>>> shortcomings > > > >>>>>> in the current DataStream API that it tries to resolve? How does > > the > > > >>>>>> current ProcessFunction functionality fit into the picture? Will > > it > > > be > > > >>>>> kept > > > >>>>>> as is or subsumed by new API? > > > >>>>>> > > > >>>>> I don't think we should create a replacement for the DataStream > API > > > >>>> unless > > > >>>>>> we have a very good reason to do so and with a proper discussion > > > about > > > >>>>> this > > > >>>>>> as Alex said. > > > >>>>> > > > >>>>> The ProcessFunction API which is targeting to replace DataStream > > API > > > is > > > >>>>> still a proposal, not a decision. Sorry for the confusion, I > should > > > >> have > > > >>>>> been more careful with my words, not giving the impression that > > this > > > is > > > >>>>> something we'll do anyway. > > > >>>>> > > > >>>>> There will be a FLIP describing the motivations and designs in > > > detail, > > > >>>> for > > > >>>>> the community to discuss and vote on. We are still working on it. > > > TBH, > > > >>>> this > > > >>>>> is not trivial and we would need more time on it. > > > >>>>> > > > >>>>> Just to quickly share some backgrounds: > > > >>>>> > > > >>>>> - We see quite some problems with the current DataStream APIs > > > >>>>> - Users are working with concrete classes rather than > > > >> interfaces, > > > >>>>> which means > > > >>>>> - Users can access methods that are designed to be used by > > > >> internal > > > >>>>> classes, even though they are annotated with > `@Internal`. > > > >> E.g., > > > >>>>> `DataStream#getTransformation`. > > > >>>>> - Changes to the non-API implementations (e.g., > > > >>>> `Transformation`) > > > >>>>> would affect the API classes (e.g., `DataStream`), which > > > >>>>> makes it hard to > > > >>>>> provide binary compatibility. > > > >>>>> - Internal classes are used as parameter / return-value of > > > >> public > > > >>>>> APIs. E.g., while `AbstractStreamOperator` is > PublicEvolving, > > > >>>>> `StreamTask` > > > >>>>> which returns from > `AbstractStreamOperator#getContainingTask` > > > is > > > >>>>> Internal. > > > >>>>> - In many cases, users are asked to extend the API classes, > > > >> rather > > > >>>>> than implementing interfaces. E.g., > `AbstractStreamOperator`. > > > >>>>> - Any changes to the base classes, even the internal > part, > > > >> may > > > >>>>> affect the behavior of the user-provided sub-classes > > > >>>>> - Users can override the behavior of the base classes > > > >>>>> - The API module `flink-streaming-java` contains non-API > > > >> classes, > > > >>>> and > > > >>>>> depends on internal modules such as `flink-runtime`, which > > > means > > > >>>>> - Changes to the internal modules may affect the API > modules, > > > >> which > > > >>>>> requires users to re-build their applications upon > > upgrading > > > >>>>> - The artifact user needs for building their application > > > >> larger > > > >>>>> than necessary. > > > >>>>> - We probably should not expose operators (e.g., > > > >>>>> `AbstractStreamOperator`) to users. Functions should be > > enough > > > >>>>> for users to > > > >>>>> define their data processing logics. Exposing > operator-level > > > >>>> concepts > > > >>>>> (e.g., mailbox thread model, checkpoint barrier alignment, > > > >> etc.) is > > > >>>>> unnecessary and limits the improvement regarding such > exposed > > > >>>>> mechanisms > > > >>>>> with compatibility considerations. > > > >>>>> - The current DataStream API seems to be a mixture of many > > > >> things, > > > >>>>> making it hard to understand especially for newcomers. It > > might > > > >> be > > > >>>>> better > > > >>>>> to re-organize it into several parts: (the taxonomy below > are > > > >> just > > > >>>> an > > > >>>>> example of the, we are still working on this) > > > >>>>> - The most fundamental stateful stream processing: > > streams, > > > >>>>> partitions / key, process functions, state, > > timeline-service > > > >>>>> - An extension for common batch-streaming unified > > functions: > > > >>>> map, > > > >>>>> flatmap, filter, agg, reduce, join, etc. > > > >>>>> - An extension for windowing supports: window, > triggering > > > >>>>> - An extension for event-time supports: event time, > > > watermark > > > >>>>> - The extensions are like short-cuts / sugars, without > > which > > > >>>> users > > > >>>>> can probably still achieve the same behavior by working > > with > > > >> the > > > >>>>> fundamental APIs, but would be a lot easier with the > > > >> extensions > > > >>>>> - The original plan was to do in-place refactors / changes > on > > > >>>>> DataStream API. Some related items are listed in this doc [2] > > > >> attached > > > >>>>> to > > > >>>>> the kicking off email [3]. Not all of the above issues are > > listed, > > > >>>>> because > > > >>>>> we haven't looked into this as deeply as now by that time. > > > >>>>> - We proposed this as a new API rather than in-place refactors > > in > > > >> the > > > >>>>> 2.0 work item list, because we realized the changes might be > too > > > >> big > > > >>>>> for an > > > >>>>> in-place change. First having a new API then gradually > retiring > > > the > > > >>>> old > > > >>>>> one > > > >>>>> would help users to smoothly migrate between them. > > > >>>>> > > > >>>>> A thorough discussion is definitely needed once the FLIP is out. > > And > > > of > > > >>>>> course it's possible that the FLIP might be rejected. Given that > we > > > are > > > >>>>> planning for release 2.0, I just feel it would be better to bring > > > this > > > >> up > > > >>>>> early even the concrete plan is not yet ready, > > > >>>>> > > > >>>>> Best, > > > >>>>> > > > >>>>> Xintong > > > >>>>> > > > >>>>> > > > >>>>> [1] > > https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 > > > >>>>> [2] > > > >>>>> > > > >>>>> > > > >>>> > > > >> > > > > > > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit?usp=sharing > > > >>>>> [3] > > https://lists.apache.org/thread/b8w5cx0qqbwzzklyn5xxf54vw9ymys1c > > > >>>>> > > > >>>>> On Tue, Jun 27, 2023 at 5:15 PM Gyula Fóra <gyf...@apache.org> > > > wrote: > > > >>>>> > > > >>>>>> Hey! > > > >>>>>> > > > >>>>>> I share the same concerns mentioned above regarding the > > > >>>> "ProcessFunction > > > >>>>>> API". > > > >>>>>> > > > >>>>>> I don't think we should create a replacement for the DataStream > > API > > > >>>>> unless > > > >>>>>> we have a very good reason to do so and with a proper discussion > > > about > > > >>>>> this > > > >>>>>> as Alex said. > > > >>>>>> > > > >>>>>> Cheers, > > > >>>>>> Gyula > > > >>>>>> > > > >>>>>> On Tue, Jun 27, 2023 at 11:03 AM Alexander Fedulov < > > > >>>>>> alexander.fedu...@gmail.com> wrote: > > > >>>>>> > > > >>>>>>> Hi Xintong, > > > >>>>>>> > > > >>>>>>> By compatibility discussion do you mean the "[DISCUSS] > FLIP-321: > > > >>>>>> Introduce > > > >>>>>>> an API deprecation process" thread [1]? > > > >>>>>>> > > > >>>>>>> I am also curious to know if the rationale behind this new API > > has > > > >>>> been > > > >>>>>>> previously discussed on the mailing list. Do we have a list of > > > >>>>>> shortcomings > > > >>>>>>> in the current DataStream API that it tries to resolve? How > does > > > the > > > >>>>>>> current ProcessFunction functionality fit into the picture? > Will > > it > > > >>>> be > > > >>>>>> kept > > > >>>>>>> as is or subsumed by new API? > > > >>>>>>> > > > >>>>>>> [1] > > > https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 > > > >>>>>>> > > > >>>>>>> Best, > > > >>>>>>> Alex > > > >>>>>>> > > > >>>>>>> On Mon, 26 Jun 2023 at 14:33, Xintong Song < > > tonysong...@gmail.com> > > > >>>>>> wrote: > > > >>>>>>>>> The ProcessFunction API item is giving me the most headaches > > > >>>>> because > > > >>>>>>> it's > > > >>>>>>>>> very unclear what it actually entails; like is it an entirely > > > >>>>>> separate > > > >>>>>>>> API > > > >>>>>>>>> to DataStream (sounds like it is!) or an extension of > > DataStream. > > > >>>>> How > > > >>>>>>>> much > > > >>>>>>>>> will it share the internals with DataStream etc.; how does it > > > >>>>> relate > > > >>>>>> to > > > >>>>>>>> the > > > >>>>>>>>> Table API (w.r.t. switching APIs / what Table API uses > > > >>>> underneath). > > > >>>>>>>> I totally understand your confusion. We started planning this > > > after > > > >>>>>>> kicking > > > >>>>>>>> off the release 2.0, so there's still a lot to be explored and > > the > > > >>>>> plan > > > >>>>>>>> keeps changing. > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> - In the beginning, we planned to do an in-place refactor > of > > > >>>>>>> DataStream > > > >>>>>>>> API, until the API migration period is proposed. > > > >>>>>>>> - Then we want to make it an entirely separate API to > > > >>>> DataStream, > > > >>>>>> and > > > >>>>>>>> listed as a must-have for release 2.0 so that we can remove > > > >>>>>> DataStream > > > >>>>>>>> once > > > >>>>>>>> it's ready. > > > >>>>>>>> - However, depending on the outcome of the API > compatibility > > > >>>>>>> discussion > > > >>>>>>>> [1], we may not be able to remove DataStream in 2.0 anyway, > > > >>>> which > > > >>>>>>> means > > > >>>>>>>> we > > > >>>>>>>> might need to re-evaluate the necessity of this item for > 2.0. > > > >>>>>>>> > > > >>>>>>>> I'd say we wait a bit longer for the compatibility discussion > > [1] > > > >>>> and > > > >>>>>>>> decide the priority for this item afterwards. > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> Best, > > > >>>>>>>> > > > >>>>>>>> Xintong > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> [1] https://lists.apache.org/list.html?dev@flink.apache.org > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> On Mon, Jun 26, 2023 at 6:00 PM Chesnay Schepler < > > > >>>> ches...@apache.org > > > >>>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>>> by-and-large I'm quite happy with the list of items. > > > >>>>>>>>> > > > >>>>>>>>> I'm curious as to why the "Disaggregated State Management" > item > > > >>>> is > > > >>>>>>> marked > > > >>>>>>>>> as a must-have; will it require changes that break something? > > > >>>> What > > > >>>>>>>> prevents > > > >>>>>>>>> it from being added in 2.1? > > > >>>>>>>>> > > > >>>>>>>>> We may want to update the Java 17 item to "Make Java 17 the > > > >>>>> default, > > > >>>>>>> drop > > > >>>>>>>>> Java 8/11". Maybe even split it into a must-have "Drop Java > 8" > > > >>>> and > > > >>>>> a > > > >>>>>>>>> nice-to-have "Drop Java 11"? > > > >>>>>>>>> > > > >>>>>>>>> "Move Calcite rules from Scala to Java": I would hope that > this > > > >>>>> would > > > >>>>>>> be > > > >>>>>>>>> an entirely internal change, and could thus be an incremental > > > >>>>> process > > > >>>>>>>>> independent of major releases. > > > >>>>>>>>> What is the actual scale of this item; how much are we > actually > > > >>>>>>>> re-writing? > > > >>>>>>>>> "Add MetricGroup#getLogicalScope": I'd raise this to a > > > >>>> must-have; i > > > >>>>>>> think > > > >>>>>>>>> I marked it down as nice-to-have only because it depends on > > > >>>> another > > > >>>>>>> item. > > > >>>>>>>>> The ProcessFunction API item is giving me the most headaches > > > >>>>> because > > > >>>>>>> it's > > > >>>>>>>>> very unclear what it actually entails; like is it an entirely > > > >>>>>> separate > > > >>>>>>>> API > > > >>>>>>>>> to DataStream (sounds like it is!) or an extension of > > DataStream. > > > >>>>> How > > > >>>>>>>> much > > > >>>>>>>>> will it share the internals with DataStream etc.; how does it > > > >>>>> relate > > > >>>>>> to > > > >>>>>>>> the > > > >>>>>>>>> Table API (w.r.t. switching APIs / what Table API uses > > > >>>> underneath). > > > >>>>>>>>> There are a few items I added as ideas which don't have a > > > >>>> priority > > > >>>>>> yet; > > > >>>>>>>>> would love to get some feedback on those. > > > >>>>>>>>> > > > >>>>>>>>> On 21/06/2023 08:41, Xintong Song wrote: > > > >>>>>>>>> > > > >>>>>>>>> Hi devs, > > > >>>>>>>>> > > > >>>>>>>>> As previously discussed in [1], we had been collecting work > > item > > > >>>>>>>> proposals > > > >>>>>>>>> for the 2.0 release until June 15th, on the wiki page [2]. > > > >>>>>>>>> > > > >>>>>>>>> - As we have passed the due date, I'd like to kindly > remind > > > >>>>>> everyone > > > >>>>>>>> *not > > > >>>>>>>>> to add / remove items directly on the wiki page*. If > needed, > > > >>>>>> please > > > >>>>>>>> post > > > >>>>>>>>> in this thread or reach out to the release managers > instead. > > > >>>>>>>>> - I've reached out to some folks for clarifications about > > > >>>> their > > > >>>>>>>>> proposals. Some of them mentioned that they can not yet > tell > > > >>>>>> whether > > > >>>>>>>> we > > > >>>>>>>>> should do an item or not, and would need more time / > > > >>>> discussions > > > >>>>>> to > > > >>>>>>>> make > > > >>>>>>>>> the decision. So I added a new symbol for items whose > > > >>>> priorities > > > >>>>>> are > > > >>>>>>>> `TBD`. > > > >>>>>>>>> Now it's time to collaboratively decide a minimum set of > > > >>>> must-have > > > >>>>>>> items. > > > >>>>>>>>> I've gone through the entire list of proposed items, and > found > > > >>>> most > > > >>>>>> of > > > >>>>>>>> them > > > >>>>>>>>> make quite much sense. So I think an online sync might not be > > > >>>>>> necessary > > > >>>>>>>> for > > > >>>>>>>>> this. I'd like to go with this DISCUSS thread, where everyone > > can > > > >>>>>>> comment > > > >>>>>>>>> on how they think the list can be improved, followed by a > VOTE > > to > > > >>>>>>>> formally > > > >>>>>>>>> make the decision. > > > >>>>>>>>> > > > >>>>>>>>> Any feedback and opinions, including but not limited to the > > > >>>>> following > > > >>>>>>>>> aspects, will be appreciated. > > > >>>>>>>>> > > > >>>>>>>>> - Important items that are missing from the list > > > >>>>>>>>> - Concerns regarding the listed items or their priorities > > > >>>>>>>>> > > > >>>>>>>>> Looking forward to your feedback. > > > >>>>>>>>> > > > >>>>>>>>> Best, > > > >>>>>>>>> > > > >>>>>>>>> Xintong > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> [1] > > > >>>> > > > >> > > > > > > https://lists.apache.org/list?dev@flink.apache.org:lte=1M:release%202.0%20status%20updates > > > >>>>>>>>> [2] > > > >>>> https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>> > > > >>>> -- > > > >>>> Best regards, > > > >>>> Sergey > > > >>>> > > > >> > > > >> > > > > > > > > > -- Best regards, Sergey
