Hi JB, On Tue, Nov 22, 2016 at 11:14 AM, Jean-Baptiste Onofré <[email protected]> wrote: > > DoFn will execute per element (with eventually a hook on StartBundle, > FinishBundle, and Teardown). It's basic the way it works in IO WriteFn: we > create the connection in StartBundle and send each element (with a batch) > to external resource. > > PTransform is maybe more flexible in case of interact with "outside" > resources. >
Probably PTransform would be a better place. I'm still pretty new to some of the Beam terms and apis. Do you have use case to be sure I understand ? Yes, Well, it's far more complex, but this question I can simplify it: We have a TensorFlow-based classifier. In our pipeline one step performs that classification of the data. Currently it's implemented as a Spark Function, because TensorFlow models can directly be embedded within pipelines using PySpark. Therefore I'm looking for the best option to move such classification process one level up in the abstraction with Beam, so I could make it portable. The first idea I'm exploring is relying on a external function (i.e., microservice) that I'd need to scale up and down independently of the pipeline. So I'm more than welcome to discuss ideas ;-) Thanks. Cheers, > On 11/22/2016 10:39 AM, Sergio Fernández wrote: > >> Hi, >> >> I'd like resume the idea to have TensorFlow-based tasks running in a Beam >> Pipeline. So far the cleaner approach I can imagine would be to have it >> running outside (Functions in GCP, Lambdas in AWS, Microservices generally >> speaking). >> >> Therefore, does the current Beam model provide the sense of a DoFn which >> actually runs externally? >> >> Thanks in advance for the feedback. >> >> Cheers, >> >> > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com > > -- > <http://www.talend.com> > <http://www.talend.com> > Sergio Fernández > Partner Technology Manager > Redlink GmbH > m: +43 6602747925 > e: <http://www.talend.com>[email protected] > w: http://redlink.co >
