+1, I like it. Thanks! On Fri, Mar 23, 2018 at 9:03 AM Ahmet Altay <al...@google.com> wrote:
> Thank you Ismaël, this looks really cool. > > On Fri, Mar 23, 2018 at 5:33 AM, Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi, >> >> it sounds like a very good extension mechanism to PTransform. >> >> +1 >> >> Regards >> JB >> >> On 03/23/2018 12:03 PM, Ismaël Mejía wrote: >> > This is a really simple proposal to add an extension with transforms >> > that package the Java Scripting API )JSR-223) [1] to allow users to >> > specialize some transforms via a scripting language. This work was >> > initially created by Romain [2] and I just took it with his >> > authorization and refined it to make it pass all the Beam validations >> > + style. I also added ValueProviders that allow users to template now >> > scripts also in Dataflow. >> > >> > Notice that Dataflow recently added something similar to create really >> > simple data movement pipelines [3], so maybe the rest of the community >> > can benefit of a similar extension (and eventually dataflow may >> > converge to this implementation). >> > >> > I hope there is interest in this extension, so far we have a >> > ScriptingParDo transform to show the idea, hopefully we can expand >> > this to other transforms. >> > >> > For those interested in more details you can check the Jira issue [4] >> > and the PR [5]. >> > >> > [1] https://www.jcp.org/en/jsr/detail?id=223 >> > [2] https://github.com/rmannibucau/beam-jsr223 >> > [3] >> https://cloud.google.com/blog/big-data/2018/03/pre-built-cloud-dataflow-templates-kiss-for-data-movement >> > [4] https://issues.apache.org/jira/browse/BEAM-3921 >> > [5} https://github.com/apache/beam/pull/4944 >> > >> >> -- >> Jean-Baptiste Onofré >> jbono...@apache.org >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> > >
