Sorry for posting on a separate thread. Lets continue the discussion here.
+1 for having URN to identify environment type. I think URN is better
than 'oneof' structure as its more flexible and forward compatible.
On Fri, Aug 31, 2018 at 9:14 AM Thomas Weise <t...@apache.org
<mailto:t...@apache.org>> wrote:
FYI the first part of support for (direct) process based job bundle
factory was merged (thanks Max!)
https://github.com/apache/beam/pull/6287
On top of that I have built a customization that runs the Python SDK
worker directly:
https://github.com/lyft/beam/blob/release-2.8.0-lyft/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/LyftProcessJobBundleFactory.java
While doing that, I thought it would be nice to support a custom
environment (along with the other options we already discussed) that
allows users to hook in their own extensions without having to do
stuff like this:
https://github.com/lyft/beam/pull/6/files#diff-b74ff692340bcae0032d119a7192624cR61
Thanks,
Thomas
On Mon, Aug 27, 2018 at 2:43 AM Robert Bradshaw <rober...@google.com
<mailto:rober...@google.com>> wrote:
On Mon, Aug 27, 2018 at 11:23 AM Maximilian Michels
<m...@apache.org <mailto:m...@apache.org>> wrote:
>
> Thanks for your proposal Henning. +1 for explicit environment
messages.
> I'm not sure how important it is to support cross-platform
pipelines. I
> can foresee future use but I wouldn't consider it essential.
One may want to execute Tensforflow pipeline segments from the
middle
of a Java pipeline, or leverage the SQL code (currently
implemented in
Java) from Python or Go. In my experience, a common pipeline
shape is
to do a significant amount of (often fairly trivial) filtering
at the
front of a pipeline, and sophisticated analysis at the end, and the
tradeoffs of execution efficiency vs. expressiveness and prototype
friendliness are different in these two halves of the same pipeline.
> However, it
> basically comes for free if we extend the existing environment
> information for ExecutableStage. The overhead, as you said,
is negligible.
+1. Also it should be noted that the runner is ideally not
restricted
to this set of environments; if it understands the URN it can use
whatever environment it finds appropriate. This could be especially
useful for optimal choice of environment to avoid unneeded fusion
barriers (e.g. the trivial Count transform will has a pair-with-one
before the GBK, and a sum-values after the GBK, and assuming those
operations are available in nearly every environment it would be
preferable to choose the variant according to what
precedes/follows it
to allow fusion (or, possibly, even embed the operation).
> Also agree that artifact staging is important even with
process-based
> execution. The execution environment might be managed
externally but we
> still want to be able to execute new pipelines without
copying over
> required artifact. That said, a first version could come without
> artifact staging.
One of the parameters passed to the script is the staging endpoint,
which it can use to do all the staging itself, so this could also be
internal to the script. I can however see wanting the ability to
stage
artifacts more cheaply (e.g. symlinks), and this is actually
also the
case with the docker environment (e.g. mount points).
> On 23.08.18 18:14, Henning Rohde wrote:
> > A process-based SDK harness does not IMO imply that the
host is fully
> > provisioned by the SDK/user and invoking the user command
line in the
> > context of the staged files is a critical aspect for it to
work. So I
> > consider staged artifact support needed. Also, I would like
to suggest
> > that we move to a concrete environment proto to crystalize
what is
> > actually being proposed. I'm not sure what activating a
virtualenv would
> > look like, for example. To start things off:
> >
> > message Environment {
> > string urn = 1;
> > bytes payload = 2;
> > }
> >
> > // urn == "beam:env:docker:v1"
> > message DockerPayload {
> > string container_image = 1; // implicitly linux_amd64.
> > }
> >
> > // urn == "beam:env:process:v1"
> > message ProcessPayload {
> > string os = 1; // "linux", "darwin", ..
> > string arch = 2; // "amd64", ..
> > string command_line = 3;
> > }
> >
> > // urn == "beam:env:external:v1"
> > // (no payload)
> >
> > A runner may support any subset and reject any unsupported
> > configuration. There are 3 kinds of environments that I
think are useful:
> > (1) docker: works as currently. Offers the most
flexibility for SDKs
> > and users, especially when the runner is outside the
control (such
> > as hosted runners). The runner starts the SDK harnesses.
> > (2) process: as discussed here. The runner starts the SDK
harnesses.
> > The semantics is that the shell commandline is invoked in a
directory
> > rooted in the staged artifacts with the container contract
arguments. It
> > is up to the user and runner deployment to ensure that it
makes sense,
> > i.e., on windows a linux binary or bash script is not
specified.
> > Executing the user command in a shell env (bash, zsh, cmd,
..) ensures
> > that paths and so on are set up:, i.e., specifying "java
-jar foo" would
> > actually work. Useful for cases where the user controls
both the SDK and
> > runner (such as locally) or when docker is not an option.
Intended to be
> > minimal and SDK/language agnostic.
> > (3) external: this is what I think Robert was alluding
to. The runner
> > does not start any SDK harnesses. Instead it waits for
user-controlled
> > SDK harnesses to connect. Useful for manually debugging SDK
code
> > (connect from code running in a debugger) or when the user
code must run
> > in a special or privileged environment. It's
runner-specific how the SDK
> > will need to connect.
> >
> > Part of the idea of placing this information in the
environment is that
> > pipelines can potentially use multiple, such as
cross-windows/linux.
> >
> > Henning
> >
> > On Thu, Aug 23, 2018 at 6:44 AM Thomas Weise
<t...@apache.org <mailto:t...@apache.org>
> > <mailto:t...@apache.org <mailto:t...@apache.org>>> wrote:
> >
> > I would see support for staging libraries as optional /
nice to have
> > since that can also be done as part of host
provisioning (i.e. in
> > the Python case a virtual environment was already setup
and just
> > needs to be activated).
> >
> > Depending on how the command that launches the harness is
> > configured, additional steps such as virtualenv
activate or setting
> > of other environment variables can be included as well.
> >
> >
> > On Thu, Aug 23, 2018 at 5:15 AM Maximilian Michels
<m...@apache.org <mailto:m...@apache.org>
> > <mailto:m...@apache.org <mailto:m...@apache.org>>> wrote:
> >
> > Just to recap:
> >
> > From this and the other thread ("Bootstraping
Beam's Job
> > Server") we
> > got sufficient evidence that process-based
execution is a
> > desired feature.
> >
> > Process-based execution as an alternative to
dockerized execution
> > https://issues.apache.org/jira/browse/BEAM-5187
> >
> > Which parts are executed as a process?
> > => The SDK harness for user code
> >
> > What configuration options are supported?
> > => Provide information about the target
architecture (OS/CPU)
> > => Staging libraries, as also supported by Docker
> > => Activating a pre-existing environment (e.g.
virutalenv)
> >
> >
> > On 23.08.18 14:13, Maximilian Michels wrote:
> > >> One thing to consider that we've talked about
in the past.
> > It might
> > >> make sense to extend the environment proto and
have the SDK be
> > >> explicit about which kinds of environment it
support
> > >
> > > +1 Encoding environment information there is a
good idea.
> > >
> > >> Seems it will create a default docker url even
if the
> > >> hardness_docker_image is set to None in
pipeline options.
> > Shall we add
> > >> another option or honor the None in this option
to support
> > the process
> > >> job?
> > >
> > > Yes, if no Docker image is set the default one
will be used.
> > Currently
> > > Docker is the only way to execute pipelines with the
> > PortableRunner. If
> > > the docker_image is not set, execution won't
succeed.
> > >
> > > On 22.08.18 22:59, Xinyu Liu wrote:
> > >> We are also interested in this Process
JobBundleFactory as
> > we are
> > >> planning to fork a process to run python sdk in
Samza
> > runner, instead
> > >> of using docker container. So this change will
be helpful to
> > us too.
> > >> On the same note, we are trying out
portable_runner.py to
> > submit a
> > >> python job. Seems it will create a default
docker url even
> > if the
> > >> hardness_docker_image is set to None in
pipeline options.
> > Shall we add
> > >> another option or honor the None in this option
to support
> > the process
> > >> job? I made some local changes right now to
walk around this.
> > >>
> > >> Thanks,
> > >> Xinyu
> > >>
> > >> On Tue, Aug 21, 2018 at 12:25 PM, Henning Rohde
> > <hero...@google.com <mailto:hero...@google.com>
<mailto:hero...@google.com <mailto:hero...@google.com>>
> > >> <mailto:hero...@google.com
<mailto:hero...@google.com> <mailto:hero...@google.com
<mailto:hero...@google.com>>>> wrote:
> > >>
> > >> By "enum" in quotes, I meant the usual open
URN style
> > pattern not an
> > >> actual enum. Sorry if that wasn't clear.
> > >>
> > >> On Tue, Aug 21, 2018 at 11:51 AM Lukasz Cwik
> > <lc...@google.com <mailto:lc...@google.com>
<mailto:lc...@google.com <mailto:lc...@google.com>>
> > >> <mailto:lc...@google.com
<mailto:lc...@google.com> <mailto:lc...@google.com
<mailto:lc...@google.com>>>> wrote:
> > >>
> > >> I would model the environment to be
more free form
> > then enums
> > >> such that we have forward looking
extensibility and
> > would
> > >> suggest to follow the same pattern we
use on
> > PTransforms (using
> > >> an URN and a URN specific payload).
Note that in
> > this case we
> > >> may want to support a list of supported
environments
> > (e.g. java,
> > >> docker, python, ...).
> > >>
> > >> On Tue, Aug 21, 2018 at 10:37 AM
Henning Rohde
> > >> <hero...@google.com
<mailto:hero...@google.com> <mailto:hero...@google.com
<mailto:hero...@google.com>>
> > <mailto:hero...@google.com
<mailto:hero...@google.com> <mailto:hero...@google.com
<mailto:hero...@google.com>>>> wrote:
> > >>
> > >> One thing to consider that we've
talked about in
> > the past.
> > >> It might make sense to extend the
environment
> > proto and have
> > >> the SDK be explicit about which
kinds of
> > environment it
> > >> supports:
> > >>
> > >>
> > >>
> >
https://github.com/apache/beam/blob/8c4f4babc0b0d55e7bddefa3f9f9ba65d21ef139/model/pipeline/src/main/proto/beam_runner_api.proto#L969
> >
> > >>
> > >>
> > >>
> >
<https://github.com/apache/beam/blob/8c4f4babc0b0d55e7bddefa3f9f9ba65d21ef139/model/pipeline/src/main/proto/beam_runner_api.proto#L969>
> >
> > >>
> > >>
> > >> This choice might impact what files
are staged
> > or what not.
> > >> Some SDKs, such as Go, provide a
compiled binary
> > and _need_
> > >> to know what the target
architecture is. Running
> > on a mac
> > >> with and without docker, say,
requires a
> > different worker in
> > >> each case. If we add an "enum", we
can also
> > easily add the
> > >> external idea where the SDK/user
starts the SDK
> > harnesses
> > >> instead of the runner. Each runner
may not
> > support all types
> > >> of environments.
> > >>
> > >> Henning
> > >>
> > >> On Tue, Aug 21, 2018 at 2:52 AM
Maximilian Michels
> > >> <m...@apache.org
<mailto:m...@apache.org> <mailto:m...@apache.org
<mailto:m...@apache.org>>
> > <mailto:m...@apache.org <mailto:m...@apache.org>
<mailto:m...@apache.org <mailto:m...@apache.org>>>> wrote:
> > >>
> > >> For reference, here is
corresponding JIRA
> > issue for this
> > >> thread:
> > >> https://issues.apache.org/jira/browse/BEAM-5187
> > >>
> > <https://issues.apache.org/jira/browse/BEAM-5187>
> > >>
> > >> On 16.08.18 11:15, Maximilian
Michels wrote:
> > >> > Makes sense to have an
option to run the
> > SDK harness
> > >> in a non-dockerized
> > >> > environment.
> > >> >
> > >> > I'm in the process of
creating a Docker
> > entry point
> > >> for Flink's
> > >> > JobServer[1]. I suppose you
would also
> > prefer to
> > >> execute that one
> > >> > standalone. We should make
sure this is
> > also an
> > >> option.
> > >> >
> > >> > [1]
> > https://issues.apache.org/jira/browse/BEAM-4130
> > >>
> > <https://issues.apache.org/jira/browse/BEAM-4130>
> > >> >
> > >> > On 16.08.18 07:42, Thomas
Weise wrote:
> > >> >> Yes, that's the proposal.
Everything
> > that would
> > >> otherwise be packaged
> > >> >> into the Docker container
would need to be
> > >> pre-installed in the host
> > >> >> environment. In the case of
Python SDK,
> > that could
> > >> simply mean a
> > >> >> (frozen) virtual
environment that was
> > setup when the
> > >> host was
> > >> >> provisioned - the SDK harness
> > process(es) will then
> > >> just utilize that.
> > >> >> Of course this flavor of
SDK harness
> > execution could
> > >> also be useful in
> > >> >> the local development
environment, where
> > right now
> > >> someone who already
> > >> >> has the Python environment
needs to also
> > install
> > >> Docker and update a
> > >> >> container to launch a
Python SDK
> > pipeline on the
> > >> Flink runner.
> > >> >>
> > >> >> On Wed, Aug 15, 2018 at
12:40 PM Daniel
> > Oliveira
> > >> <danolive...@google.com
<mailto:danolive...@google.com>
> > <mailto:danolive...@google.com
<mailto:danolive...@google.com>> <mailto:danolive...@google.com
<mailto:danolive...@google.com>
> > <mailto:danolive...@google.com
<mailto:danolive...@google.com>>>
> > >> >>
<mailto:danolive...@google.com <mailto:danolive...@google.com>
> > <mailto:danolive...@google.com
<mailto:danolive...@google.com>>
> > >> <mailto:danolive...@google.com
<mailto:danolive...@google.com>
> > <mailto:danolive...@google.com
<mailto:danolive...@google.com>>>>> wrote:
> > >> >>
> > >> >> I just want to clarify
that I
> > understand this
> > >> correctly since I'm
> > >> >> not that familiar with
the details
> > behind all
> > >> these execution
> > >> >> environments yet. Is
the proposal
> > to create a
> > >> new JobBundleFactory
> > >> >> that instead of using
Docker to
> > create the
> > >> environment that the new
> > >> >> processes will execute
in, this
> > >> JobBundleFactory would execute the
> > >> >> new processes directly
in the host
> > environment?
> > >> So in practice if I
> > >> >> ran a pipeline with this
> > JobBundleFactory the
> > >> SDK Harness and Runner
> > >> >> Harness would both be
executing
> > directly on my
> > >> machine and would
> > >> >> depend on me having the
> > dependencies already
> > >> present on my machine?
> > >> >>
> > >> >> On Mon, Aug 13, 2018
at 5:50 PM
> > Ankur Goenka
> > >> <goe...@google.com
<mailto:goe...@google.com>
> > <mailto:goe...@google.com
<mailto:goe...@google.com>> <mailto:goe...@google.com
<mailto:goe...@google.com>
> > <mailto:goe...@google.com <mailto:goe...@google.com>>>
> > >> >>
<mailto:goe...@google.com <mailto:goe...@google.com>
> > <mailto:goe...@google.com <mailto:goe...@google.com>>
> > >> <mailto:goe...@google.com
<mailto:goe...@google.com>
> > <mailto:goe...@google.com
<mailto:goe...@google.com>>>>> wrote:
> > >> >>
> > >> >> Thanks for
starting the
> > discussion. I will
> > >> be happy to help.
> > >> >> I agree, we should
have pluggable
> > >> SDKHarness environment Factory.
> > >> >> We can register
multiple
> > Environment
> > >> factory using service
> > >> >> registry and use the
> > PipelineOption to pick
> > >> the right one on per
> > >> >> job basis.
> > >> >>
> > >> >> There are a couple
of things
> > which are
> > >> require to setup before
> > >> >> launching the process.
> > >> >>
> > >> >> * Setting up the
environment
> > as done in
> > >> boot.go [4]
> > >> >> * Retrieving and
putting the
> > artifacts in
> > >> the right location.
> > >> >>
> > >> >> You can probably
leverage
> > boot.go code to
> > >> setup the environment.
> > >> >>
> > >> >> Also, it will be
useful to
> > enumerate pros
> > >> and cons of different
> > >> >> Environments to
help users
> > choose the right
> > >> one.
> > >> >>
> > >> >>
> > >> >> On Mon, Aug 6,
2018 at 4:50 PM
> > Thomas Weise
> > >> <t...@apache.org
<mailto:t...@apache.org> <mailto:t...@apache.org
<mailto:t...@apache.org>>
> > <mailto:t...@apache.org <mailto:t...@apache.org>
<mailto:t...@apache.org <mailto:t...@apache.org>>>
> > >> >>
<mailto:t...@apache.org <mailto:t...@apache.org>
> > <mailto:t...@apache.org <mailto:t...@apache.org>>
> > >> <mailto:t...@apache.org
<mailto:t...@apache.org>
> > <mailto:t...@apache.org <mailto:t...@apache.org>>>>>
wrote:
> > >> >>
> > >> >> Hi,
> > >> >>
> > >> >> Currently the
portable
> > Flink runner
> > >> only works with SDK
> > >> >> Docker
containers for execution
> > >> (DockerJobBundleFactory,
> > >> >> besides an
in-process
> > (embedded)
> > >> factory option for testing
> > >> >> [1]). I'm
considering
> > adding another
> > >> out of process
> > >> >> JobBundleFactory
> > implementation that
> > >> directly forks the
> > >> >> processes on
the task
> > manager host,
> > >> eliminating the need for
> > >> >> Docker. This
would work
> > reasonably well
> > >> in environments
> > >> >> where the
dependencies (in
> > this case
> > >> Python) can easily be
> > >> >> tied into the host
> > deployment (also
> > >> within an application
> > >> >> specific
Kubernetes pod).
> > >> >>
> > >> >> There was
already some
> > discussion about
> > >> alternative
> > >> >> JobBundleFactory
> > implementation in [2].
> > >> There is also a JIRA
> > >> >> to make the
bundle factory
> > pluggable
> > >> [3], pending
> > >> >> availability
of runner
> > level options.
> > >> >>
> > >> >> For a
> > "ProcessBundleFactory", in
> > >> addition to the Python
> > >> >> dependencies the
> > environment would also
> > >> need to have the Go
> > >> >> boot
executable [4] (or a
> > substitute
> > >> thereof) to perform the
> > >> >> harness
initialization.
> > >> >>
> > >> >> Is anyone else
interested
> > in this SDK
> > >> execution option or
> > >> >> has already
investigated an
> > alternative
> > >> implementation?
> > >> >>
> > >> >> Thanks,
> > >> >> Thomas
> > >> >>
> > >> >> [1]
> > >> >>
> > >>
> > >>
> >
https://github.com/apache/beam/blob/7958a379b0a37a89edc3a6ae4b5bc82fda41fcd6/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java#L83
> >
> > >>
> > >>
> > >>
> >
<https://github.com/apache/beam/blob/7958a379b0a37a89edc3a6ae4b5bc82fda41fcd6/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java#L83>
> >
> > >>
> > >> >>
> > >> >> [2]
> > >> >>
> > >>
> > >>
> >
https://lists.apache.org/thread.html/d6b6fde764796de31996db9bb5f9de3e7aaf0ab29b99d0adb52ac508@%3Cdev.beam.apache.org%3E
> >
> > >>
> > >>
> > >>
> >
<https://lists.apache.org/thread.html/d6b6fde764796de31996db9bb5f9de3e7aaf0ab29b99d0adb52ac508@%3Cdev.beam.apache.org%3E>
> >
> > >>
> > >> >>
> > >> >> [3]
> > >> https://issues.apache.org/jira/browse/BEAM-4819
> > >>
> > <https://issues.apache.org/jira/browse/BEAM-4819>
> > >> >>
> > >> >> [4]
> > >>
> > >>
> >
https://github.com/apache/beam/blob/master/sdks/python/container/boot.go
> > >>
> > >>
> >
<https://github.com/apache/beam/blob/master/sdks/python/container/boot.go>
> >
> > >>
> > >> >>
> > >>
> > >> -- Max
> > >>
> > >>
> > >
> >
> > --
> > Max
> >
>
> --
> Max
