@Maciek I saw that I missed replying to your question:
> Could you please remind what was the conclusion of discussion on upgrading Scala to 2.12.15/16? > https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I couldn't find any follow-up vote? There is a vote thread, but that never got enough votes. See https://lists.apache.org/thread/l93l5qqr5n2oty3r2bjsz3ks3tjf1655 > If it's acceptable to break binary compatibility by such an upgrade, then upgrading to JDK17 before 2.0 will be doable? I'm not sure, because I don't think a discussion and vote has been made yet if upgrading JDK17 can/will be done in a Flink 1.0 release or if it requires a 2.0 release. It was mentioned in the original discussion thread on dropping Java 8 support within 2/3 releases, but if I recall correctly there was no discussion yet on when Java 17 support would be added [1]. Best regards, Martijn [1] https://lists.apache.org/thread/0fwo7nwzy51gck4vxhyfnbnttd4jycpx On Wed, Oct 5, 2022 at 6:58 AM Gaël Renoux <gael.ren...@datadome.co> wrote: > > I'm curious what target Scala versions people are currently interested > in. > > I would've expected that everyone wants to migrate to Scala 3, for which > several wrapper projects around Flink already exist > > The Scala 3 tooling is still subpar (we're using IntelliJ), so I'm not > sure I would move my team to Scala 3 right now (I'm currently toying with > it on a personal project). In addition, moving to Scala 3 is not completely > free - you have to do some rewrites, and developers will need some > adaptation time. Scala 2.13 is another thing entirely, we've wanted to > migrate for a long while. > > On Wed, Oct 5, 2022 at 12:53 PM Chesnay Schepler <ches...@apache.org> > wrote: > >> > It's possible that for the sake of the Scala API, we would occasionally >> require some changes in the Java API. As long as those changes are not >> detrimental to Java users, they should be considered. >> >> That's exactly the model we're trying to get to. Don't fix scala-specific >> issues with scala code, but rather on the Java side as much as possible >> which could also benefit other JVM languages (e.g., Kotlin). >> >> > A question regarding the Flink wrapper: would it be possible to keep it >> under the Flink project's umbrella? Or does it need to be a completely >> separate structure? I'm not aware of the full organizational implications >> of this, I'm afraid. >> >> Technically it can be under the Flink umbrella, but then Flink would >> still be (at least for a while) be the bottleneck because we'd have to >> review any changes coming in. >> That would only improve once several new committers were added to take >> care of this project. >> (In the end you'd just split Flink and the Scala API _codebases_, but >> achieve little else) >> >> > And if that is what it takes to move beyond Scala 2.12.7… This has been >> a big pain point for us. >> >> I'm curious what target Scala versions people are currently interested in. >> I would've expected that everyone wants to migrate to Scala 3, for which >> several wrapper projects around Flink already exist. >> >> On 05/10/2022 12:35, Gaël Renoux wrote: >> >> Hello everyone, >> >> I've already answered a bit on Twitter, I'll develop my thoughts a bit >> here. For context, my company (DataDome) has a significant codebase on >> Scala Flink (around 110K LOC), having been using it since 2017. I myself am >> an enthusiastic Scala developer (I don't think I'd like moving back to >> Java) >> >> Given that, I think separating the Scala support from Flink is actually a >> good move long term. We would then have a full-Java Flink, and a separate >> Scala wrapper. It would help a lot in solving the skills issue: Flink >> maintainers would no longer need to be fluent in Scala, and maintainers of >> the Scala wrapper would not need a deep knowledge of Flink's inner >> workings, just the API would be sufficient. And if that is what it takes to >> move beyond Scala 2.12.7… This has been a big pain point for us. >> >> I'm not too worried about finding contributors for the Scala wrapper. >> Within my company, we have developed additional wrappers and extension >> methods (for parts where we felt the Flink Scala API was insufficient), and >> we've been looking at ways we could contribute back. What held us back was >> our lack of knowledge of the full Flink environment (we're only using the >> Scala Datastream API). I don't think we're the only ones in that situation. >> One major point, though, is that Flink developers would not be completely >> rid of us ;-) It's possible that for the sake of the Scala API, we would >> occasionally require some changes in the Java API. As long as those changes >> are not detrimental to Java users, they should be considered. >> >> A question regarding the Flink wrapper: would it be possible to keep it >> under the Flink project's umbrella? Or does it need to be a completely >> separate structure? I'm not aware of the full organizational implications >> of this, I'm afraid. >> >> Finally, the hard part would be the migration to the new version. My >> dream solution would be to have the existing Scala API be entirely >> converted into a Scala wrapper over the Java API. That way, migration would >> be pretty minimal: add a dependency, change the imports for the Scala API, >> and we're done. However, even starting from the existing flink4s project, >> that's still quite a lot of work. So, more realistically, I'd settle for at >> least a partial implementation. We would have some broken code that we >> could fix, but at the very least I'd like the basic DataStream functions >> (process, uid, name…) to be available. >> >> Thanks for all the work that went into making Flink what it is! >> >> >> Gaël Renoux - Lead R&D Engineer >> E - gael.ren...@datadome.co >> W - www.datadome.co >> >> >> >> On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <m...@touk.pl> wrote: >> >>> Hi Martin, >>> >>> Could you please remind what was the conclusion of discussion on >>> upgrading Scala to 2.12.15/16? >>> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I >>> couldn't find any follow-up vote? >>> >>> If it's acceptable to break binary compatibility by such an upgrade, >>> then upgrading to JDK17 before 2.0 will be doable? >>> >>> >>> thanks, >>> >>> maciek >>> >>> >>> On 04.10.2022 18:21, Martijn Visser wrote: >>> >>> Hi Yaroslav, >>> >>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a >>> prerequisite, we would have to break compatibility already since Scala >>> 2.12.7 doesn't compile on Java 17 [1]. >>> >>> Given that we can only remove Scala APIs with the next major Flink (2.0) >>> version, would that still impact you a lot? I do imagine that if we get to >>> a Flink 2.0 version there would be more breaking involved anyway. The >>> biggest consequence of deprecating support for Scala in Flink 1.x would be >>> that new APIs would only be available in Java, but since these don't exist >>> yet there would be no refactoring involved. I can imagine that we might >>> change something in an existing API, but that would have certain >>> compatibility guarantees already (depending if it's >>> Public/PublicEvolving/Experimental). If a change would happen there, I >>> think it would be smaller refactoring. >>> >>> Best regards, >>> >>> Martijn >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-25000 >>> >>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <yaros...@goldsky.com> >>> wrote: >>> >>>> Hi Martijn, >>>> >>>> As a Scala user, this change would affect me a lot and I'm not looking >>>> forward to rewriting my codebase, and it's not even a very large one :) >>>> >>>> I'd like to suggest supporting Java 17 as a prerequisite ( >>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch >>>> expressions and records could simplify the migration quite a bit. Would you >>>> consider adding it to the FLIP? >>>> >>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <j...@ververica.com> wrote: >>>> >>>>> Hi Martijn, >>>>> >>>>> Thanks for bringing this up. It is generally a great idea, so +1. >>>>> >>>>> Since both scala extension projects mentioned in the FLIP are still >>>>> very young and I don't think they will attract more scala developers as >>>>> Flink could just because they are external projects. It will be a big >>>>> issue >>>>> for users who have to rewrite their large codebases. Those users should be >>>>> aware of the effort from now on and would better not count on those scala >>>>> extension projects and prepare their migration plan before Flink 2.0. >>>>> >>>>> Best regards, >>>>> Jing >>>>> >>>>> >>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser < >>>>> martijnvis...@apache.org> wrote: >>>>> >>>>>> Hi Marton, >>>>>> >>>>>> You're making a good point, I originally wanted to include already >>>>>> the User mailing list to get their feedback but forgot to do so. I'll do >>>>>> some more outreach via other channels as well. >>>>>> >>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala >>>>>> API support in a future version of Flink. Your feedback on this topic is >>>>>> very much appreciated. >>>>>> >>>>>> Regarding the large Scala codebase for Flink, a potential alternative >>>>>> could be to have a wrapper for all Java APIs that makes them available as >>>>>> Scala APIs. However, this still requires Scala maintainers and I don't >>>>>> think that we currently have those in our community. The easiest solution >>>>>> for them would be to use the Java APIs directly. Yes it would involve >>>>>> work, >>>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so >>>>>> there's still time for that :) >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Martijn >>>>>> >>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi < >>>>>> balassi.mar...@gmail.com> wrote: >>>>>> >>>>>>> Hi Martjin, >>>>>>> >>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala >>>>>>> poses >>>>>>> considerable maintenance overhead and key improvements (like 2.13 or >>>>>>> 2.12.8 >>>>>>> supports) are hanging stale. With that said before we make this move >>>>>>> we >>>>>>> should attempt to understand the userbase affected. >>>>>>> A quick Slack and user mailing list search does return quite a bit of >>>>>>> results for scala (admittedly a cursory look at them suggest that >>>>>>> many of >>>>>>> them have to do with missing features in Scala that exist in Java or >>>>>>> Scala >>>>>>> versions). I would love to see some polls on this topic, we could >>>>>>> also use >>>>>>> the Flink twitter handle to ask the community about this. >>>>>>> >>>>>>> I am aware of users having large existing Scala codebases for Flink. >>>>>>> This >>>>>>> move would pose a very large effort on them, as they would need to >>>>>>> rewrite >>>>>>> much of their existing code. What are the alternatives in your >>>>>>> opinion, >>>>>>> Martjin? >>>>>>> >>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser < >>>>>>> martijnvis...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>> > Hi everyone, >>>>>>> > >>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and >>>>>>> remove >>>>>>> > Scala API support. Please take a look at >>>>>>> > >>>>>>> > >>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support >>>>>>> > and provide your feedback. >>>>>>> > >>>>>>> > Best regards, >>>>>>> > >>>>>>> > Martijn >>>>>>> > https://twitter.com/MartijnVisser82 >>>>>>> > https://github.com/MartijnVisser >>>>>>> > >>>>>>> >>>>>> >>
