Nice, it is great to see a good amount of support and enthusiasm on this. I want just to remind that the whole idea and code donation comes from Romain Manni-Bucau. I just did some ‘mise-en-forme’ plus ValueProviders. All credit to Romain!
Eugene thanks a lot for the feedback. I would like to get this initial version in quickly and iterate incrementally on the design and implementation of the features you propose afterwards, there are a lot of good ideas, thanks for sharing those. I will create a doc so we can iterate on the design of the different points. All feature requests and ideas are welcome so please to the other people in the community, feel free to add those here for discussion then we can bootstrap a better design document. On Fri, Mar 23, 2018 at 8:32 PM, Thomas Weise <t...@apache.org> wrote: > +1, nice! > > On Fri, Mar 23, 2018 at 4:03 AM, Ismaël Mejía <ieme...@gmail.com> wrote: >> >> This is a really simple proposal to add an extension with transforms >> that package the Java Scripting API )JSR-223) [1] to allow users to >> specialize some transforms via a scripting language. This work was >> initially created by Romain [2] and I just took it with his >> authorization and refined it to make it pass all the Beam validations >> + style. I also added ValueProviders that allow users to template now >> scripts also in Dataflow. >> >> Notice that Dataflow recently added something similar to create really >> simple data movement pipelines [3], so maybe the rest of the community >> can benefit of a similar extension (and eventually dataflow may >> converge to this implementation). >> >> I hope there is interest in this extension, so far we have a >> ScriptingParDo transform to show the idea, hopefully we can expand >> this to other transforms. >> >> For those interested in more details you can check the Jira issue [4] >> and the PR [5]. >> >> [1] https://www.jcp.org/en/jsr/detail?id=223 >> [2] https://github.com/rmannibucau/beam-jsr223 >> [3] >> https://cloud.google.com/blog/big-data/2018/03/pre-built-cloud-dataflow-templates-kiss-for-data-movement >> [4] https://issues.apache.org/jira/browse/BEAM-3921 >> [5} https://github.com/apache/beam/pull/4944 > >
