Thanks Max, I took a look at making this change directly to the scripts. I was initially thinking about making a separate script whose only responsibility is to run the command in the foreground, so that the flink-daemon.sh could delegate to this script. I didn't get very far into though, mostly trying to find a nice way to share the logging configuration and passing of parameters.
While doing this, it did occur to me that if the process is run in the foreground, the logging should be written to standard out. This ended up being slightly hairy, at least in the job manager case, because the web UI client expects the standard out of the job manager to be written to a file, but I seem to have gotten things to work. I just wanted to mention this detail in case it's the convention for things running in the foreground, in which case it may need to be implemented alongside foregrounding the process. Thanks, Brian On Thu, Dec 3, 2015 at 3:55 AM, Maximilian Michels <m...@apache.org> wrote: > I think the way supervisor is used in the Docker scripts is a bit hacky. > It is simply started in the foreground and does nothing. Supervisor is > actually a really nice utility to start processes in Docker containers and > monitor them. > > Nevertheless, supervisor also expects commands to stay in the foreground. > A common way to work around this, is to create a script which monitors the > daemon process' pid. Thinking about this, I think we could actually add the > foreground functionality directly in the jobmanager / taskmanager shell > script like you suggested. > > In the meantime, you could also use a simple script like this: > > #!/usr/bin/env bash > # daemonize job manager > ./bin/jobmanager start cluster > # wait until process goes down > wait $! > > Cheers, > Max > > On Wed, Dec 2, 2015 at 7:16 PM, Brian Chhun <brian.ch...@getbraintree.com> > wrote: > >> Thanks, I'm basing the things I'm doing based on what I see there. One >> thing that's not clear to me in that example is why supervisor is used to >> keep the container alive, rather than using some simpler means. It doesn't >> look like it's been configured to supervise anything. >> >> On Wed, Dec 2, 2015 at 11:44 AM, Maximilian Michels <m...@apache.org> >> wrote: >> >>> Have you looked at >>> https://github.com/apache/flink/tree/master/flink-contrib/docker-flink >>> ? This demonstrates how to use Flink with Docker. In particular it >>> states: "Images [..] run Supervisor to stay alive when running >>> containers." >>> >>> Have a look at flink/config-flink.sh. >>> >>> Cheers, >>> Max >>> >>> On Wed, Dec 2, 2015 at 6:29 PM, Brian Chhun >>> <brian.ch...@getbraintree.com> wrote: >>> > Yep, I think this makes sense. I'm currently patching the >>> flink-daemon.sh >>> > script to remove the `&`, but I don't think it's a very robust >>> solution, >>> > particularly when this script changes across versions of Flink. I'm >>> very new >>> > to Docker, but the resources I've found indicates that the process >>> must run >>> > in the foreground, though people seem to get around it with some hacks. >>> > >>> > When I have some time, I can look into refactoring some parts of the >>> scripts >>> > so that it can be started in the foreground. >>> > >>> > Thanks, >>> > Brian >>> > >>> > On Wed, Dec 2, 2015 at 3:22 AM, Maximilian Michels <m...@apache.org> >>> wrote: >>> >> >>> >> Hi Brian, >>> >> >>> >> I don't recall Docker requires commands to run in the foreground. >>> Still, >>> >> if that is your requirement, simply remove the "&" at the end of this >>> line >>> >> in flink-daemon.sh: >>> >> >>> >> $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" >>> -classpath >>> >> "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" >>> >> ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null & >>> >> >>> >> Cheers, >>> >> Max >>> >> >>> >> On Wed, Dec 2, 2015 at 9:26 AM, Till Rohrmann <trohrm...@apache.org> >>> >> wrote: >>> >>> >>> >>> Hi Brian, >>> >>> >>> >>> as far as I know this is at the moment not possible with our scripts. >>> >>> However it should be relatively easy to add by simply executing the >>> Java >>> >>> command in flink-daemon.sh in the foreground. Do you want to add >>> this? >>> >>> >>> >>> Cheers, >>> >>> Till >>> >>> >>> >>> On Dec 1, 2015 9:40 PM, "Brian Chhun" <brian.ch...@getbraintree.com> >>> >>> wrote: >>> >>>> >>> >>>> Hi All, >>> >>>> >>> >>>> Is it possible to include a command line flag for starting job and >>> task >>> >>>> managers in the foreground? Currently, `bin/jobmanager.sh` and >>> >>>> `bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts >>> these >>> >>>> things in the background. I'd like to execute these commands inside >>> a docker >>> >>>> container, but it's expected that the process is running in the >>> foreground. >>> >>>> I think it might be useful to have it run in the foreground so that >>> it can >>> >>>> be hooked into some process supervisors. Any suggestions are >>> appreciated. >>> >>>> >>> >>>> >>> >>>> Thanks, >>> >>>> Brian >>> >> >>> >> >>> > >>> >> >> >
