Added comment access to the doc link. On Wed, Aug 7, 2019 at 3:00 AM Robert Bradshaw <[email protected]> wrote:
> On Wed, Aug 7, 2019 at 6:20 AM Thomas Weise <[email protected]> wrote: > > > > Hi Kyle, > > > > [document doesn't have comments enabled currently] > > > > As noted, worker deployment is an open question. I believe pipeline > submission and worker execution need to be considered together for a > complete deployment story. The idea of creating a self containing jar file > is interesting, but there are trade-offs: > > > > * The pipeline construction code itself may need access to cluster > resources. In such cases the jar file cannot be created offline. > > Could you elaborate? > > > * For k8s deployment, a container image with the SDK and application > code is required for the worker. The jar file (which is really a derived > artifact) would need to be built in addition to the container image. > > Yes. For standard use, a vanilla released Beam published SDK container > + staged artifacts should be sufficient. > > > * To build such jar file, the user would need a build environment with > job server and application code. Do we want to make that assumption? > > Actually, it's probably much easier than that. A jar file is just a > zip file with a standard structure, to which one can easily add (data) > files without having a full build environment. The (pre-compiled) main > class would know how to read this data to construct the pipeline and > kick off the job just like any other Flink job. > > > The document that I had shared discusses options for pipeline > submission. It might be interesting to explore if your proposal for > building such a jar can be integrated or if you have other comments? > > > > Thomas > > > > > > > > On Tue, Aug 6, 2019 at 5:03 PM Kyle Weaver <[email protected]> wrote: > >> > >> Hi all, > >> > >> Following up on discussion about portable Beam on Flink on Kubernetes > [1], I have drafted a short document on how I propose we bundle portable > Beam applications into jars that can be run on OSS runners, similar to > Dataflow templates (but without the actual template part, at least for the > first iteration). It's pretty straightforward, but I thought I would > broadcast it here in case anyone is interested. > >> > >> > https://docs.google.com/document/d/1kj_9JWxGWOmSGeZ5hbLVDXSTv-zBrx4kQRqOq85RYD4/edit# > >> > >> [1] > https://lists.apache.org/thread.html/a12dd939c4af254694481796bc08b05bb1321cfaadd1a79cd3866584@%3Cdev.beam.apache.org%3E > >> > >> Kyle Weaver | Software Engineer | github.com/ibzib | > [email protected] > -- Kyle Weaver | Software Engineer | github.com/ibzib | [email protected]
