Hi,

What do you think about collecting some of the feedback from the community at Beam Summit last week? Here's what I've come across:


* The Kubernetes / Docker Story

Multiple users reported that they would like a Beam-Kubernetes story. What is the best way to deploy Beam with Kubernetes? Will there be built-in support?

Especially with regards to the portability, there are some unsolved problems, e.g. how to start Beam containerized and bootstrap the SDK Harness container from within a container? For local testing with the JobServer we support that via mounting the Docker socket, but this will be too fragile in production scenarios. Now that we have process-based execution, we could just use that inside the main container.

Deployment is a very important topic for users and we should try to reduce complexity as much as possible.

* External SDKs / Scio

Users have asked why Scio is not part of the main repository. Generally, I don't think that has to be the case, same for the Runners which are not part of the main repo. However, it does raise the question, what will be the future model for maintaining SDKs/IOs/Runners? How do we ensure easy development and a consistent quality of internal/external components?

* Documenting Timers & State

These two have excellent blog posts but are not part of the official documentation. Since they are part of the model, it would be good to eventually update the docs.

* Better Debuggability of pipelines

Even a simple WordCount in Beam leads to a quite complex Flink execution graph (due to the the involved I/O logic). How can we make pipelines easier to understand? Will we provide a way to visualize the architecture of high-level Beam pipelines? If so, do we provide a way to gain insight into how it is mapped to the Runner execution model? Users would like to have more insight.

* Current Roadmap

This was asked in the context of portability. By the end of the year we should have at least the FlinkRunner in a ready state, with the rest following up. There are a lot of others threads in Beam. The newsletter is a great way to keep up with the project development.


Looking forward to any other points you might have.

Best,
Max

Reply via email to