Obviously, it depends on your opinion, Gabor. What I said it just my mind :D
On Sun, Jan 23, 2022 at 5:17 PM Bertty Contreras <[email protected]> wrote: > Hi Gabor, > > Thank you so much for your effort, we really appreciate it as community :D > > I was thinking about you contributing what you have done at it, and then > we can create issues with all the missing parts, and when some come or have > free time can take the issue and work on it :D. to have it done :D. > > About the Scala 3, we can work in the design and split in different issues > then as the previous idea, we will have several issues, but we will be > working before the platforms come out with the scala 3, wayang will be > ready :D. > > Personally I believe that having the issue could motivate more contributor > :D > > Best regards, > Bertty > > On Sun, Jan 23, 2022 at 2:25 PM Gábor E. Gévay <[email protected]> wrote: > >> Hi All, >> >> I would like to also write here my thoughts about the Wayang-Emma >> integration, so there is a public record of it :) >> >> I tried to do the integration in September (while I had some idle time >> in the PhD waiting for reviewers), but a big technical problem is that >> Emma relies on Scala macros. Emma was written using Scala 2 macros, >> but Scala 2 macros are very messy. Scala 3 would be better, as it has >> a completely redesigned macro system, but Scala 3 is not yet properly >> supported by Flink and Spark. So I think it would be good to wait >> until Scala 3 is more widely supported (at least by Flink and/or >> Spark), and then create an Emma-like API for Wayang using Scala 3 >> macros. But this will be a much bigger effort than just integrating >> the current Emma, and unfortunately I probably won't have enough free >> capacity for this. >> >> Btw. I still think that, in the long term, Wayang could benefit from >> the API design approach of Emma, i.e. using macros to look at the >> syntax tree of the user code (to look inside UDFs, or examine loops, >> etc.). Without macros, i.e., if an API relies only on classes and >> their method calls, API design tends to get too closely coupled to the >> underlying execution system's internals. An example is how doWhile >> needs to be explicitly present as a method call in the current Wayang >> Scala API in order to enable translating a loop to an efficient Flink >> loop. Flink can efficiently execute loops as cyclic dataflows, but >> this execution-level detail comes up to Wayang's API level because >> without macros it's impossible to translate imperative control flow to >> an efficient Flink dataflow. Macros would make it possible to compile >> from imperative control flow (i.e., normal Scala loops instead of >> building loops with method calls) to efficient cyclic dataflows, i.e., >> decouple the API design from the execution details. >> >> So I hope Flink and Spark get that Scala 3 support properly working >> soon, and then it will be possible to use Scala 3 macros in Wayang's >> API design :D >> >> Best, >> Gábor >> >
