Hello everyone, I've already answered a bit on Twitter, I'll develop my thoughts a bit here. For context, my company (DataDome) has a significant codebase on Scala Flink (around 110K LOC), having been using it since 2017. I myself am an enthusiastic Scala developer (I don't think I'd like moving back to Java)
Given that, I think separating the Scala support from Flink is actually a good move long term. We would then have a full-Java Flink, and a separate Scala wrapper. It would help a lot in solving the skills issue: Flink maintainers would no longer need to be fluent in Scala, and maintainers of the Scala wrapper would not need a deep knowledge of Flink's inner workings, just the API would be sufficient. And if that is what it takes to move beyond Scala 2.12.7… This has been a big pain point for us. I'm not too worried about finding contributors for the Scala wrapper. Within my company, we have developed additional wrappers and extension methods (for parts where we felt the Flink Scala API was insufficient), and we've been looking at ways we could contribute back. What held us back was our lack of knowledge of the full Flink environment (we're only using the Scala Datastream API). I don't think we're the only ones in that situation. One major point, though, is that Flink developers would not be completely rid of us ;-) It's possible that for the sake of the Scala API, we would occasionally require some changes in the Java API. As long as those changes are not detrimental to Java users, they should be considered. A question regarding the Flink wrapper: would it be possible to keep it under the Flink project's umbrella? Or does it need to be a completely separate structure? I'm not aware of the full organizational implications of this, I'm afraid. Finally, the hard part would be the migration to the new version. My dream solution would be to have the existing Scala API be entirely converted into a Scala wrapper over the Java API. That way, migration would be pretty minimal: add a dependency, change the imports for the Scala API, and we're done. However, even starting from the existing flink4s project, that's still quite a lot of work. So, more realistically, I'd settle for at least a partial implementation. We would have some broken code that we could fix, but at the very least I'd like the basic DataStream functions (process, uid, name…) to be available. Thanks for all the work that went into making Flink what it is! Gaël Renoux - Lead R&D Engineer E - gael.ren...@datadome.co W - www.datadome.co On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <m...@touk.pl> wrote: > Hi Martin, > > Could you please remind what was the conclusion of discussion on upgrading > Scala to 2.12.15/16? > https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I > couldn't find any follow-up vote? > > If it's acceptable to break binary compatibility by such an upgrade, then > upgrading to JDK17 before 2.0 will be doable? > > > thanks, > > maciek > > > On 04.10.2022 18:21, Martijn Visser wrote: > > Hi Yaroslav, > > Thanks for the feedback, that's much appreciated! Regarding Java 17 as a > prerequisite, we would have to break compatibility already since Scala > 2.12.7 doesn't compile on Java 17 [1]. > > Given that we can only remove Scala APIs with the next major Flink (2.0) > version, would that still impact you a lot? I do imagine that if we get to > a Flink 2.0 version there would be more breaking involved anyway. The > biggest consequence of deprecating support for Scala in Flink 1.x would be > that new APIs would only be available in Java, but since these don't exist > yet there would be no refactoring involved. I can imagine that we might > change something in an existing API, but that would have certain > compatibility guarantees already (depending if it's > Public/PublicEvolving/Experimental). If a change would happen there, I > think it would be smaller refactoring. > > Best regards, > > Martijn > > [1] https://issues.apache.org/jira/browse/FLINK-25000 > > On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <yaros...@goldsky.com> > wrote: > >> Hi Martijn, >> >> As a Scala user, this change would affect me a lot and I'm not looking >> forward to rewriting my codebase, and it's not even a very large one :) >> >> I'd like to suggest supporting Java 17 as a prerequisite ( >> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch >> expressions and records could simplify the migration quite a bit. Would you >> consider adding it to the FLIP? >> >> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <j...@ververica.com> wrote: >> >>> Hi Martijn, >>> >>> Thanks for bringing this up. It is generally a great idea, so +1. >>> >>> Since both scala extension projects mentioned in the FLIP are still very >>> young and I don't think they will attract more scala developers as Flink >>> could just because they are external projects. It will be a big issue for >>> users who have to rewrite their large codebases. Those users should be >>> aware of the effort from now on and would better not count on those scala >>> extension projects and prepare their migration plan before Flink 2.0. >>> >>> Best regards, >>> Jing >>> >>> >>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <martijnvis...@apache.org> >>> wrote: >>> >>>> Hi Marton, >>>> >>>> You're making a good point, I originally wanted to include already the >>>> User mailing list to get their feedback but forgot to do so. I'll do some >>>> more outreach via other channels as well. >>>> >>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API >>>> support in a future version of Flink. Your feedback on this topic is very >>>> much appreciated. >>>> >>>> Regarding the large Scala codebase for Flink, a potential alternative >>>> could be to have a wrapper for all Java APIs that makes them available as >>>> Scala APIs. However, this still requires Scala maintainers and I don't >>>> think that we currently have those in our community. The easiest solution >>>> for them would be to use the Java APIs directly. Yes it would involve work, >>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so >>>> there's still time for that :) >>>> >>>> Best regards, >>>> >>>> Martijn >>>> >>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi <balassi.mar...@gmail.com> >>>> wrote: >>>> >>>>> Hi Martjin, >>>>> >>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala >>>>> poses >>>>> considerable maintenance overhead and key improvements (like 2.13 or >>>>> 2.12.8 >>>>> supports) are hanging stale. With that said before we make this move we >>>>> should attempt to understand the userbase affected. >>>>> A quick Slack and user mailing list search does return quite a bit of >>>>> results for scala (admittedly a cursory look at them suggest that many >>>>> of >>>>> them have to do with missing features in Scala that exist in Java or >>>>> Scala >>>>> versions). I would love to see some polls on this topic, we could also >>>>> use >>>>> the Flink twitter handle to ask the community about this. >>>>> >>>>> I am aware of users having large existing Scala codebases for Flink. >>>>> This >>>>> move would pose a very large effort on them, as they would need to >>>>> rewrite >>>>> much of their existing code. What are the alternatives in your opinion, >>>>> Martjin? >>>>> >>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser < >>>>> martijnvis...@apache.org> >>>>> wrote: >>>>> >>>>> > Hi everyone, >>>>> > >>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and >>>>> remove >>>>> > Scala API support. Please take a look at >>>>> > >>>>> > >>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support >>>>> > and provide your feedback. >>>>> > >>>>> > Best regards, >>>>> > >>>>> > Martijn >>>>> > https://twitter.com/MartijnVisser82 >>>>> > https://github.com/MartijnVisser >>>>> > >>>>> >>>>
