Hi Andrey, thanks a lot for this proposal. The variety of Docker files in the project has been causing quite some confusion.
For the entrypoint, have you considered to also allow setting configuration via environment variables as in "docker run -e FLINK_REST_BIN_PORT=8081 ..."? This is quite common and more flexible, e.g. it makes it very easy to pass values of Kubernetes Secrets into the Flink configuration. With respect to logging, I would opt to keep this very basic and to only support logging to the console (maybe with a fix for the web user interface). For everything else, users can easily build their own images based on library/flink (provide the dependencies, change the logging configuration). Cheers, Konstantin On Thu, Mar 5, 2020 at 11:01 AM Yang Wang <danrtsey...@gmail.com> wrote: > Hi Andrey, > > > Thanks for driving this significant FLIP. From the user ML, we could also > know there are > many users running Flink in container environment. Then the docker image > will be the > very basic requirement. Just as you say, we should provide a unified place > for all various > usage(e.g. session, job, native k8s, swarm, etc.). > > > > About docker utils > > I really like the idea to provide some utils for the docker file and entry > point. The > `flink_docker_utils` will help to build the image easier. I am not sure > about the > `flink_docker_utils start_jobmaster`. Do you mean when we build a docker > image, we > need to add `RUN flink_docker_utils start_jobmaster` in the docker file? > Why do we need this? > > > > About docker entry point > > I agree with you that the docker entry point could more powerful with more > functionality. > Mostly, it is about to override the config options. If we support dynamic > properties, i think > it is more convenient for users without any learning curve. > `docker run flink session_jobmanager -D rest.bind-port=8081` > > > > About the logging > > Updating the `log4j-console.properties` to support multiple appender is a > better option. > Currently, the native K8s is suggesting users to debug the logs in this > way[1]. However, > there is also some problems. The stderr and stdout of JM/TM processes > could not be > forwarded to the docker container console. > > > [1]. > https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html#log-files > > > Best, > Yang > > > > > Andrey Zagrebin <azagre...@apache.org> 于2020年3月4日周三 下午5:34写道： > >> Hi All, >> >> If you have ever touched the docker topic in Flink, you >> probably noticed that we have multiple places in docs and repos which >> address its various concerns. >> >> We have prepared a FLIP [1] to simplify the perception of docker topic in >> Flink by users. It mostly advocates for an approach of extending official >> Flink image from the docker hub. For convenience, it can come with a set >> of >> bash utilities and documented examples of their usage. The utilities allow >> to: >> >> - run the docker image in various modes (single job, session master, >> task manager etc) >> - customise the extending Dockerfile >> - and its entry point >> >> Eventually, the FLIP suggests to remove all other user facing Dockerfiles >> and building scripts from Flink repo, move all docker docs to >> apache/flink-docker and adjust existing docker use cases to refer to this >> new approach (mostly Kubernetes now). >> >> The first contributed version of Flink docker integration also contained >> example and docs for the integration with Bluemix in IBM cloud. We also >> suggest to maintain it outside of Flink repository (cc Markus Müller). >> >> Thanks, >> Andrey >> >> [1] >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification >> > -- Konstantin Knauf | Head of Product +49 160 91394525 Follow us @VervericaData Ververica <https://www.ververica.com/> -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Tony) Cheng
