Hi All, I would like to also write here my thoughts about the Wayang-Emma integration, so there is a public record of it :)
I tried to do the integration in September (while I had some idle time in the PhD waiting for reviewers), but a big technical problem is that Emma relies on Scala macros. Emma was written using Scala 2 macros, but Scala 2 macros are very messy. Scala 3 would be better, as it has a completely redesigned macro system, but Scala 3 is not yet properly supported by Flink and Spark. So I think it would be good to wait until Scala 3 is more widely supported (at least by Flink and/or Spark), and then create an Emma-like API for Wayang using Scala 3 macros. But this will be a much bigger effort than just integrating the current Emma, and unfortunately I probably won't have enough free capacity for this. Btw. I still think that, in the long term, Wayang could benefit from the API design approach of Emma, i.e. using macros to look at the syntax tree of the user code (to look inside UDFs, or examine loops, etc.). Without macros, i.e., if an API relies only on classes and their method calls, API design tends to get too closely coupled to the underlying execution system's internals. An example is how doWhile needs to be explicitly present as a method call in the current Wayang Scala API in order to enable translating a loop to an efficient Flink loop. Flink can efficiently execute loops as cyclic dataflows, but this execution-level detail comes up to Wayang's API level because without macros it's impossible to translate imperative control flow to an efficient Flink dataflow. Macros would make it possible to compile from imperative control flow (i.e., normal Scala loops instead of building loops with method calls) to efficient cyclic dataflows, i.e., decouple the API design from the execution details. So I hope Flink and Spark get that Scala 3 support properly working soon, and then it will be possible to use Scala 3 macros in Wayang's API design :D Best, Gábor
