Hey, Just adding my 2c here - we've been pretty ok with creating a custom job factory that listens on JVM shutdown hooks and notify the ThreadJob to stop executing. This will trigger the RunLoop to stop executing and call the close method on all running task instances. You could see an example for it here: https://gist.github.com/anonymous/6ceb7ecbf80775380725
I haven't had a chance to investigate how the YARN based job factory handles it, but if you have any experience on running it on top of mesos/docker it would be very interesting to hear Thanks, Dotan On Wed, Jan 14, 2015 at 12:15 AM, Chris Riccomini < [email protected]> wrote: > Hey Zach, > > Ah, I see what you're saying. Yes, to implement this as a bolt-on on top > of the framework is somewhat cumbersome. > > The "right" thing to do is to put the addShutdownHook call into RunLoop, > itself, and have it change the shutdownNow variable when it gets a > SIGTERM. That requires a patch, though. :) In the meantime, what you have > seems like it should work, but I agree that it's a bit hacky. > > Cheers, > Chris > > On 1/13/15 1:57 PM, "Zach Cox" <[email protected]> wrote: > > >I created a jira ticket for this: > >https://issues.apache.org/jira/browse/SAMZA-506 > > > >On Tue, Jan 13, 2015 at 3:49 PM, Zach Cox <[email protected]> wrote: > > > >> Hi Chris, > >> > >> Here's the main idea of the SIGTERM shutdown hook approach that we're > >> trying out: https://gist.github.com/zcox/6ec8910bd3f18e36c1a2 > >> > >> We use our own main class to run either the job or the container > >>depending > >> on args, and this main is the ENTRYPOINT of our Docker container, > >>instead > >> of using the standard run-job.sh and run-container.sh scripts. So this > >>is > >> an easy place to add the shutdown hook. Not sure how reusable that > >>would be > >> for others... > >> > >> By "not in a very clean way" I meant that the shutdown hook needs to > >>block > >> jvm shutdown until SamzaContainer has gone through its entire list of > >> shutdown calls, and the only way I could find for the shutdown hook to > >>know > >> this is finished is the shutdownIsComplete flag, set true after > >> SamzaContainer.safeMain() returns, which feels a bit jank, but it seems > >> to work. > >> > >> Thanks, > >> Zach > >> > >> > >> On Tue, Jan 13, 2015 at 2:34 PM, Chris Riccomini < > >> [email protected]> wrote: > >> > >>> Hey Zach, > >>> > >>> Honestly, we just tell YARN to kill everything off. As you said, this > >>> means there is no clean shutdown, but we've lived with thus far. > >>> > >>> Some folks have also used a control message, as you've said, and had > >>>their > >>> task call coordinator.shutdown(). Others have used a JMX switch to do > >>>the > >>> same. > >>> > >>> I agree that catching a SIGTERM is a little cleaner. Can you open a > >>>JIRA > >>> for this? > >>> > >>> Basically, all the solutions you've described are possible. Only the > >>> SIGTERM trap hasn't been implemented. > >>> > >>> > I can sorta make this work, but not in a very clean way. > >>> > >>> > >>> Curious: what's not clean? > >>> > >>> Cheers, > >>> Chris > >>> > >>> On 1/13/15 11:21 AM, "Zach Cox" <[email protected]> wrote: > >>> > >>> >Hi - just wondering what other folks do when a Samza job needs to be > >>> >shutdown purposefully, such as during a deployment. Seems like in this > >>> >case > >>> >you'd want all of the job's Samza containers to cleanly stop > >>>processing > >>> >messages, commit offsets/changelog, etc instead of just SIGKILL the > >>>jvms > >>> >which will result in out-of-date checkpoints, etc. > >>> > > >>> >I noticed TaskCoordinator has a shutdown method, which the > >>>StreamTasks in > >>> >a > >>> >container can use to signal a clean shutdown. Does everyone use this, > >>> with > >>> >some kind of shutdown message sent to the StreamTasks using e.g. jmx > >>>or a > >>> >kafka topic, as in this thread? > >>> > > >>> > >>> > http://www.mail-archive.com/dev%40samza.incubator.apache.org/msg01248.ht > >>>ml > >>> > > >>> >We're actively developing Samza support for Mesos, so maybe this is > >>> >specific to that world, but would it make sense for a jvm shutdown > >>>hook > >>> in > >>> >the container to catch a SIGTERM and somehow signal to the container > >>>that > >>> >it should shutdown? I can sorta make this work, but not in a very > >>>clean > >>> >way. > >>> > > >>> >Just curious what others are doing about graceful shutdown. > >>> > > >>> >Thanks, > >>> >Zach > >>> > >>> > >> > >
