Hi Gabor, Thank you so much for your effort, we really appreciate it as community :D
I was thinking about you contributing what you have done at it, and then we can create issues with all the missing parts, and when some come or have free time can take the issue and work on it :D. to have it done :D. About the Scala 3, we can work in the design and split in different issues then as the previous idea, we will have several issues, but we will be working before the platforms come out with the scala 3, wayang will be ready :D. Personally I believe that having the issue could motivate more contributor :D Best regards, Bertty On Sun, Jan 23, 2022 at 2:25 PM Gábor E. Gévay <[email protected]> wrote: > Hi All, > > I would like to also write here my thoughts about the Wayang-Emma > integration, so there is a public record of it :) > > I tried to do the integration in September (while I had some idle time > in the PhD waiting for reviewers), but a big technical problem is that > Emma relies on Scala macros. Emma was written using Scala 2 macros, > but Scala 2 macros are very messy. Scala 3 would be better, as it has > a completely redesigned macro system, but Scala 3 is not yet properly > supported by Flink and Spark. So I think it would be good to wait > until Scala 3 is more widely supported (at least by Flink and/or > Spark), and then create an Emma-like API for Wayang using Scala 3 > macros. But this will be a much bigger effort than just integrating > the current Emma, and unfortunately I probably won't have enough free > capacity for this. > > Btw. I still think that, in the long term, Wayang could benefit from > the API design approach of Emma, i.e. using macros to look at the > syntax tree of the user code (to look inside UDFs, or examine loops, > etc.). Without macros, i.e., if an API relies only on classes and > their method calls, API design tends to get too closely coupled to the > underlying execution system's internals. An example is how doWhile > needs to be explicitly present as a method call in the current Wayang > Scala API in order to enable translating a loop to an efficient Flink > loop. Flink can efficiently execute loops as cyclic dataflows, but > this execution-level detail comes up to Wayang's API level because > without macros it's impossible to translate imperative control flow to > an efficient Flink dataflow. Macros would make it possible to compile > from imperative control flow (i.e., normal Scala loops instead of > building loops with method calls) to efficient cyclic dataflows, i.e., > decouple the API design from the execution details. > > So I hope Flink and Spark get that Scala 3 support properly working > soon, and then it will be possible to use Scala 3 macros in Wayang's > API design :D > > Best, > Gábor >
