Thank you Ismaël, this looks really cool. On Fri, Mar 23, 2018 at 5:33 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote:
> Hi, > > it sounds like a very good extension mechanism to PTransform. > > +1 > > Regards > JB > > On 03/23/2018 12:03 PM, Ismaël Mejía wrote: > > This is a really simple proposal to add an extension with transforms > > that package the Java Scripting API )JSR-223) [1] to allow users to > > specialize some transforms via a scripting language. This work was > > initially created by Romain [2] and I just took it with his > > authorization and refined it to make it pass all the Beam validations > > + style. I also added ValueProviders that allow users to template now > > scripts also in Dataflow. > > > > Notice that Dataflow recently added something similar to create really > > simple data movement pipelines [3], so maybe the rest of the community > > can benefit of a similar extension (and eventually dataflow may > > converge to this implementation). > > > > I hope there is interest in this extension, so far we have a > > ScriptingParDo transform to show the idea, hopefully we can expand > > this to other transforms. > > > > For those interested in more details you can check the Jira issue [4] > > and the PR [5]. > > > > [1] https://www.jcp.org/en/jsr/detail?id=223 > > [2] https://github.com/rmannibucau/beam-jsr223 > > [3] https://cloud.google.com/blog/big-data/2018/03/pre-built- > cloud-dataflow-templates-kiss-for-data-movement > > [4] https://issues.apache.org/jira/browse/BEAM-3921 > > [5} https://github.com/apache/beam/pull/4944 > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
