Option #3 "external" would fit the Kubernetes use case we discussed a while
ago also. Container(s) can be part of the same pod and need to find the
runner.
There is another option: "embedded". When the SDK is Java and the runner
Flink (or all the other OSS runners), then harness can (optionally) run
embedded in the same JVM.
Thanks,
Thomas
On Thu, Aug 23, 2018 at 9:14 AM Henning Rohde <hero...@google.com> wrote:
> A process-based SDK harness does not IMO imply that the host is fully
> provisioned by the SDK/user and invoking the user command line in the
> context of the staged files is a critical aspect for it to work. So I
> consider staged artifact support needed. Also, I would like to suggest that
> we move to a concrete environment proto to crystalize what is actually
> being proposed. I'm not sure what activating a virtualenv would look like,
> for example. To start things off:
>
> message Environment {
> string urn = 1;
> bytes payload = 2;
> }
>
> // urn == "beam:env:docker:v1"
> message DockerPayload {
> string container_image = 1; // implicitly linux_amd64.
> }
>
> // urn == "beam:env:process:v1"
> message ProcessPayload {
> string os = 1; // "linux", "darwin", ..
> string arch = 2; // "amd64", ..
> string command_line = 3;
> }
>
> // urn == "beam:env:external:v1"
> // (no payload)
>
> A runner may support any subset and reject any unsupported configuration.
> There are 3 kinds of environments that I think are useful:
> (1) docker: works as currently. Offers the most flexibility for SDKs and
> users, especially when the runner is outside the control (such as hosted
> runners). The runner starts the SDK harnesses.
> (2) process: as discussed here. The runner starts the SDK harnesses. The
> semantics is that the shell commandline is invoked in a directory rooted in
> the staged artifacts with the container contract arguments. It is up to the
> user and runner deployment to ensure that it makes sense, i.e., on windows
> a linux binary or bash script is not specified. Executing the user command
> in a shell env (bash, zsh, cmd, ..) ensures that paths and so on are set
> up:, i.e., specifying "java -jar foo" would actually work. Useful for cases
> where the user controls both the SDK and runner (such as locally) or when
> docker is not an option. Intended to be minimal and SDK/language agnostic.
> (3) external: this is what I think Robert was alluding to. The runner
> does not start any SDK harnesses. Instead it waits for user-controlled SDK
> harnesses to connect. Useful for manually debugging SDK code (connect from
> code running in a debugger) or when the user code must run in a special or
> privileged environment. It's runner-specific how the SDK will need to
> connect.
>
> Part of the idea of placing this information in the environment is that
> pipelines can potentially use multiple, such as cross-windows/linux.
>
> Henning
>
> On Thu, Aug 23, 2018 at 6:44 AM Thomas Weise <t...@apache.org> wrote:
>
>> I would see support for staging libraries as optional / nice to have
>> since that can also be done as part of host provisioning (i.e. in the
>> Python case a virtual environment was already setup and just needs to be
>> activated).
>>
>> Depending on how the command that launches the harness is configured,
>> additional steps such as virtualenv activate or setting of other
>> environment variables can be included as well.
>>
>>
>> On Thu, Aug 23, 2018 at 5:15 AM Maximilian Michels <m...@apache.org>
>> wrote:
>>
>>> Just to recap:
>>>
>>> From this and the other thread ("Bootstraping Beam's Job Server") we
>>> got sufficient evidence that process-based execution is a desired
>>> feature.
>>>
>>> Process-based execution as an alternative to dockerized execution
>>> https://issues.apache.org/jira/browse/BEAM-5187
>>>
>>> Which parts are executed as a process?
>>> => The SDK harness for user code
>>>
>>> What configuration options are supported?
>>> => Provide information about the target architecture (OS/CPU)
>>> => Staging libraries, as also supported by Docker
>>> => Activating a pre-existing environment (e.g. virutalenv)
>>>
>>>
>>> On 23.08.18 14:13, Maximilian Michels wrote:
>>> >> One thing to consider that we've talked about in the past. It might
>>> >> make sense to extend the environment proto and have the SDK be
>>> >> explicit about which kinds of environment it support
>>> >
>>> > +1 Encoding environment information there is a good idea.
>>> >
>>> >> Seems it will create a default docker url even if the
>>> >> hardness_docker_image is set to None in pipeline options. Shall we
>>> add
>>> >> another option or honor the None in this option to support the
>>> process
>>> >> job?
>>> >
>>> > Yes, if no Docker image is set the default one will be used. Currently
>>> > Docker is the only way to execute pipelines with the PortableRunner.
>>> If
>>> > the docker_image is not set, execution won't succeed.
>>> >
>>> > On 22.08.18 22:59, Xinyu Liu wrote:
>>> >> We are also interested in this Process JobBundleFactory as we are
>>> >> planning to fork a process to run python sdk in Samza runner, instead
>>> >> of using docker container. So this change will be helpful to us too.
>>> >> On the same note, we are trying out portable_runner.py to submit a
>>> >> python job. Seems it will create a default docker url even if the
>>> >> hardness_docker_image is set to None in pipeline options. Shall we
>>> add
>>> >> another option or honor the None in this option to support the
>>> process
>>> >> job? I made some local changes right now to walk around this.
>>> >>
>>> >> Thanks,
>>> >> Xinyu
>>> >>
>>> >> On Tue, Aug 21, 2018 at 12:25 PM, Henning Rohde <hero...@google.com
>>> >> <mailto:hero...@google.com>> wrote:
>>> >>
>>> >> By "enum" in quotes, I meant the usual open URN style pattern not
>>> an
>>> >> actual enum. Sorry if that wasn't clear.
>>> >>
>>> >> On Tue, Aug 21, 2018 at 11:51 AM Lukasz Cwik <lc...@google.com
>>> >> <mailto:lc...@google.com>> wrote:
>>> >>
>>> >> I would model the environment to be more free form then enums
>>> >> such that we have forward looking extensibility and would
>>> >> suggest to follow the same pattern we use on PTransforms
>>> (using
>>> >> an URN and a URN specific payload). Note that in this case we
>>> >> may want to support a list of supported environments (e.g.
>>> java,
>>> >> docker, python, ...).
>>> >>
>>> >> On Tue, Aug 21, 2018 at 10:37 AM Henning Rohde
>>> >> <hero...@google.com <mailto:hero...@google.com>> wrote:
>>> >>
>>> >> One thing to consider that we've talked about in the past.
>>> >> It might make sense to extend the environment proto and
>>> have
>>> >> the SDK be explicit about which kinds of environment it
>>> >> supports:
>>> >>
>>> >>
>>> >>
>>> https://github.com/apache/beam/blob/8c4f4babc0b0d55e7bddefa3f9f9ba65d21ef139/model/pipeline/src/main/proto/beam_runner_api.proto#L969
>>> >>
>>> >>
>>> >> <
>>> https://github.com/apache/beam/blob/8c4f4babc0b0d55e7bddefa3f9f9ba65d21ef139/model/pipeline/src/main/proto/beam_runner_api.proto#L969>
>>>
>>> >>
>>> >>
>>> >> This choice might impact what files are staged or what
>>> not.
>>> >> Some SDKs, such as Go, provide a compiled binary and
>>> _need_
>>> >> to know what the target architecture is. Running on a mac
>>> >> with and without docker, say, requires a different worker
>>> in
>>> >> each case. If we add an "enum", we can also easily add the
>>> >> external idea where the SDK/user starts the SDK harnesses
>>> >> instead of the runner. Each runner may not support all
>>> types
>>> >> of environments.
>>> >>
>>> >> Henning
>>> >>
>>> >> On Tue, Aug 21, 2018 at 2:52 AM Maximilian Michels
>>> >> <m...@apache.org <mailto:m...@apache.org>> wrote:
>>> >>
>>> >> For reference, here is corresponding JIRA issue for
>>> this
>>> >> thread:
>>> >> https://issues.apache.org/jira/browse/BEAM-5187
>>> >> <https://issues.apache.org/jira/browse/BEAM-5187>
>>> >>
>>> >> On 16.08.18 11:15, Maximilian Michels wrote:
>>> >> > Makes sense to have an option to run the SDK
>>> harness
>>> >> in a non-dockerized
>>> >> > environment.
>>> >> >
>>> >> > I'm in the process of creating a Docker entry point
>>> >> for Flink's
>>> >> > JobServer[1]. I suppose you would also prefer to
>>> >> execute that one
>>> >> > standalone. We should make sure this is also an
>>> >> option.
>>> >> >
>>> >> > [1]
>>> https://issues.apache.org/jira/browse/BEAM-4130
>>> >> <https://issues.apache.org/jira/browse/BEAM-4130>
>>> >> >
>>> >> > On 16.08.18 07:42, Thomas Weise wrote:
>>> >> >> Yes, that's the proposal. Everything that would
>>> >> otherwise be packaged
>>> >> >> into the Docker container would need to be
>>> >> pre-installed in the host
>>> >> >> environment. In the case of Python SDK, that could
>>> >> simply mean a
>>> >> >> (frozen) virtual environment that was setup when
>>> the
>>> >> host was
>>> >> >> provisioned - the SDK harness process(es) will
>>> then
>>> >> just utilize that.
>>> >> >> Of course this flavor of SDK harness execution
>>> could
>>> >> also be useful in
>>> >> >> the local development environment, where right now
>>> >> someone who already
>>> >> >> has the Python environment needs to also install
>>> >> Docker and update a
>>> >> >> container to launch a Python SDK pipeline on the
>>> >> Flink runner.
>>> >> >>
>>> >> >> On Wed, Aug 15, 2018 at 12:40 PM Daniel Oliveira
>>> >> <danolive...@google.com <mailto:
>>> danolive...@google.com>
>>> >> >> <mailto:danolive...@google.com
>>> >> <mailto:danolive...@google.com>>> wrote:
>>> >> >>
>>> >> >> I just want to clarify that I understand this
>>> >> correctly since I'm
>>> >> >> not that familiar with the details behind all
>>> >> these execution
>>> >> >> environments yet. Is the proposal to create a
>>> >> new JobBundleFactory
>>> >> >> that instead of using Docker to create the
>>> >> environment that the new
>>> >> >> processes will execute in, this
>>> >> JobBundleFactory would execute the
>>> >> >> new processes directly in the host
>>> environment?
>>> >> So in practice if I
>>> >> >> ran a pipeline with this JobBundleFactory the
>>> >> SDK Harness and Runner
>>> >> >> Harness would both be executing directly on
>>> my
>>> >> machine and would
>>> >> >> depend on me having the dependencies already
>>> >> present on my machine?
>>> >> >>
>>> >> >> On Mon, Aug 13, 2018 at 5:50 PM Ankur Goenka
>>> >> <goe...@google.com <mailto:goe...@google.com>
>>> >> >> <mailto:goe...@google.com
>>> >> <mailto:goe...@google.com>>> wrote:
>>> >> >>
>>> >> >> Thanks for starting the discussion. I
>>> will
>>> >> be happy to help.
>>> >> >> I agree, we should have pluggable
>>> >> SDKHarness environment Factory.
>>> >> >> We can register multiple Environment
>>> >> factory using service
>>> >> >> registry and use the PipelineOption to
>>> pick
>>> >> the right one on per
>>> >> >> job basis.
>>> >> >>
>>> >> >> There are a couple of things which are
>>> >> require to setup before
>>> >> >> launching the process.
>>> >> >>
>>> >> >> * Setting up the environment as done in
>>> >> boot.go [4]
>>> >> >> * Retrieving and putting the artifacts
>>> in
>>> >> the right location.
>>> >> >>
>>> >> >> You can probably leverage boot.go code to
>>> >> setup the environment.
>>> >> >>
>>> >> >> Also, it will be useful to enumerate pros
>>> >> and cons of different
>>> >> >> Environments to help users choose the
>>> right
>>> >> one.
>>> >> >>
>>> >> >>
>>> >> >> On Mon, Aug 6, 2018 at 4:50 PM Thomas
>>> Weise
>>> >> <t...@apache.org <mailto:t...@apache.org>
>>> >> >> <mailto:t...@apache.org
>>> >> <mailto:t...@apache.org>>> wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> Currently the portable Flink runner
>>> >> only works with SDK
>>> >> >> Docker containers for execution
>>> >> (DockerJobBundleFactory,
>>> >> >> besides an in-process (embedded)
>>> >> factory option for testing
>>> >> >> [1]). I'm considering adding another
>>> >> out of process
>>> >> >> JobBundleFactory implementation that
>>> >> directly forks the
>>> >> >> processes on the task manager host,
>>> >> eliminating the need for
>>> >> >> Docker. This would work reasonably
>>> well
>>> >> in environments
>>> >> >> where the dependencies (in this case
>>> >> Python) can easily be
>>> >> >> tied into the host deployment (also
>>> >> within an application
>>> >> >> specific Kubernetes pod).
>>> >> >>
>>> >> >> There was already some discussion
>>> about
>>> >> alternative
>>> >> >> JobBundleFactory implementation in
>>> [2].
>>> >> There is also a JIRA
>>> >> >> to make the bundle factory pluggable
>>> >> [3], pending
>>> >> >> availability of runner level options.
>>> >> >>
>>> >> >> For a "ProcessBundleFactory", in
>>> >> addition to the Python
>>> >> >> dependencies the environment would
>>> also
>>> >> need to have the Go
>>> >> >> boot executable [4] (or a substitute
>>> >> thereof) to perform the
>>> >> >> harness initialization.
>>> >> >>
>>> >> >> Is anyone else interested in this SDK
>>> >> execution option or
>>> >> >> has already investigated an
>>> alternative
>>> >> implementation?
>>> >> >>
>>> >> >> Thanks,
>>> >> >> Thomas
>>> >> >>
>>> >> >> [1]
>>> >> >>
>>> >>
>>> >>
>>> https://github.com/apache/beam/blob/7958a379b0a37a89edc3a6ae4b5bc82fda41fcd6/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java#L83
>>> >>
>>> >>
>>> >> <
>>> https://github.com/apache/beam/blob/7958a379b0a37a89edc3a6ae4b5bc82fda41fcd6/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java#L83>
>>>
>>> >>
>>> >> >>
>>> >> >> [2]
>>> >> >>
>>> >>
>>> >>
>>> https://lists.apache.org/thread.html/d6b6fde764796de31996db9bb5f9de3e7aaf0ab29b99d0adb52ac508@%3Cdev.beam.apache.org%3E
>>> >>
>>> >>
>>> >> <
>>> https://lists.apache.org/thread.html/d6b6fde764796de31996db9bb5f9de3e7aaf0ab29b99d0adb52ac508@%3Cdev.beam.apache.org%3E>
>>>
>>> >>
>>> >> >>
>>> >> >> [3]
>>> >> https://issues.apache.org/jira/browse/BEAM-4819
>>> >> <https://issues.apache.org/jira/browse/BEAM-4819>
>>> >> >>
>>> >> >> [4]
>>> >>
>>> >>
>>> https://github.com/apache/beam/blob/master/sdks/python/container/boot.go
>>> >>
>>> >> <
>>> https://github.com/apache/beam/blob/master/sdks/python/container/boot.go>
>>>
>>> >>
>>> >> >>
>>> >>
>>> >> -- Max
>>> >>
>>> >>
>>> >
>>>
>>> --
>>> Max
>>>
>>
