>> Option 3) would be to map in the docker binary and socket to allow >> the containerized Flink job server to start "sibling" containers on >> the host. > >Do you mean packaging Docker inside the Job Server container and >mounting /var/run/docker.sock from the host inside the container? That >looks like a bit of a hack but for testing it could be fine.
Basically, yes, although I would also map in the docker binary itself to ensure compatibility with the host. It's not something I would suggest for production use -- just for local jobs. It would allow the Go SDK to just work OOB, for example. The process-based scenario can be a configuration feature of each SDK/runner. I see that as a useful complement to dockerized SDKs, although the onus is then on the runner/user to ensure the environment is adequate for the SDK(s) used in the job. The main appeal of docker is precisely to not have that requirement, but for some deployments it is reasonable. Henning On Mon, Aug 20, 2018 at 3:00 PM Maximilian Michels <m...@apache.org> wrote: > Thanks for your suggestions. Please see below. > > > Option 3) would be to map in the docker binary and socket to allow > > the containerized Flink job server to start "sibling" containers on > > the host. > > Do you mean packaging Docker inside the Job Server container and > mounting /var/run/docker.sock from the host inside the container? That > looks like a bit of a hack but for testing it could be fine. > > > notably, if the runner supports auto-scaling or similar non-trivial > > configurations, that would be difficult to manage from the SDK side. > > You're right, it would be unfortunate if the SDK would have to deal with > spinning up SDK harness/backend containers. For non-trivial > configurations it would probably require an extended protocol. > > > Option 4) We are also thinking about adding process based SDKHarness. > > This will avoid docker in docker scenario. > > Actually, I had started implementing a process-based SDK harness but > figured it might be impractical because it doubles the execution path > for UDF code and potentially doesn't work with custom dependencies. > > > Process based SDKHarness also has other applications and might be > > desirable in some of the production use cases. > > True. Some users might want something more lightweight. >
