I also added this as option for pipeline submission to the k8s discussion: https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.iov21d695qx5
On Thu, Aug 8, 2019 at 4:21 PM Thomas Weise <t...@apache.org> wrote: > Hi Kyle, > > It might also be useful to have the option to just output the proto and > artifacts, as alternative to the jar file. > > For the Flink entry point we would need to allow for the job server to be > used as a library. It would probably not be too hard to have the Flink job > constructed via the context execution environment, which would require no > changes on the Flink side. > > Thanks, > Thomas > > > On Thu, Aug 8, 2019 at 9:52 AM Kyle Weaver <kcwea...@google.com> wrote: > >> Re Javaless/serverless solution: >> I take it this would probably mean that we would construct the jar >> directly from the SDK. There are advantages to this: full separation of >> Python and Java environments, no need for a job server, and likely a >> simpler implementation, since we'd no longer have to work within the >> constraints of the existing job server infrastructure. The only downside I >> can think of is the additional cost of implementing/maintaining jar >> creation code in each SDK, but that cost may be acceptable if it's simple >> enough. >> >> Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com >> >> >> On Thu, Aug 8, 2019 at 9:31 AM Thomas Weise <t...@apache.org> wrote: >> >>> >>> >>> On Thu, Aug 8, 2019 at 8:29 AM Robert Bradshaw <rober...@google.com> >>> wrote: >>> >>>> > Before assembling the jar, the job server runs to create the >>>> ingredients. That requires the (matching) Java environment on the Python >>>> developers machine. >>>> >>>> We can run the job server and have it create the jar (and if we keep >>>> the job server running we can use it to interact with the running >>>> job). However, if the jar layout is simple enough, there's no need to >>>> even build it from Java. >>>> >>>> Taken to the extreme, this is a one-shot, jar-based JobService API. We >>>> choose a standard layout of where to put the pipeline description and >>>> artifacts, and can "augment" an existing jar (that has a >>>> runner-specific main class whose entry point knows how to read this >>>> data to kick off a pipeline as if it were a users driver code) into >>>> one that has a portable pipeline packaged into it for submission to a >>>> cluster. >>>> >>> >>> It would be nice if the Python developer doesn't have to run anything >>> Java at all. >>> >>> As we just discussed offline, this could be accomplished by including >>> the proto that is produced by the SDK into the pre-existing jar. >>> >>> And if the jar has an entry point that creates the Flink job in the >>> prescribed manner [1], it can be directly submitted to the Flink REST API. >>> That would allow for Java free client. >>> >>> [1] >>> https://lists.apache.org/thread.html/6db869c53816f4e2917949a7c6992c2b90856d7d639d7f2e1cd13768@%3Cdev.flink.apache.org%3E >>> >>>
