I created a jira ticket for this: https://issues.apache.org/jira/browse/SAMZA-506
On Tue, Jan 13, 2015 at 3:49 PM, Zach Cox <[email protected]> wrote: > Hi Chris, > > Here's the main idea of the SIGTERM shutdown hook approach that we're > trying out: https://gist.github.com/zcox/6ec8910bd3f18e36c1a2 > > We use our own main class to run either the job or the container depending > on args, and this main is the ENTRYPOINT of our Docker container, instead > of using the standard run-job.sh and run-container.sh scripts. So this is > an easy place to add the shutdown hook. Not sure how reusable that would be > for others... > > By "not in a very clean way" I meant that the shutdown hook needs to block > jvm shutdown until SamzaContainer has gone through its entire list of > shutdown calls, and the only way I could find for the shutdown hook to know > this is finished is the shutdownIsComplete flag, set true after > SamzaContainer.safeMain() returns, which feels a bit jank, but it seems > to work. > > Thanks, > Zach > > > On Tue, Jan 13, 2015 at 2:34 PM, Chris Riccomini < > [email protected]> wrote: > >> Hey Zach, >> >> Honestly, we just tell YARN to kill everything off. As you said, this >> means there is no clean shutdown, but we've lived with thus far. >> >> Some folks have also used a control message, as you've said, and had their >> task call coordinator.shutdown(). Others have used a JMX switch to do the >> same. >> >> I agree that catching a SIGTERM is a little cleaner. Can you open a JIRA >> for this? >> >> Basically, all the solutions you've described are possible. Only the >> SIGTERM trap hasn't been implemented. >> >> > I can sorta make this work, but not in a very clean way. >> >> >> Curious: what's not clean? >> >> Cheers, >> Chris >> >> On 1/13/15 11:21 AM, "Zach Cox" <[email protected]> wrote: >> >> >Hi - just wondering what other folks do when a Samza job needs to be >> >shutdown purposefully, such as during a deployment. Seems like in this >> >case >> >you'd want all of the job's Samza containers to cleanly stop processing >> >messages, commit offsets/changelog, etc instead of just SIGKILL the jvms >> >which will result in out-of-date checkpoints, etc. >> > >> >I noticed TaskCoordinator has a shutdown method, which the StreamTasks in >> >a >> >container can use to signal a clean shutdown. Does everyone use this, >> with >> >some kind of shutdown message sent to the StreamTasks using e.g. jmx or a >> >kafka topic, as in this thread? >> > >> http://www.mail-archive.com/dev%40samza.incubator.apache.org/msg01248.html >> > >> >We're actively developing Samza support for Mesos, so maybe this is >> >specific to that world, but would it make sense for a jvm shutdown hook >> in >> >the container to catch a SIGTERM and somehow signal to the container that >> >it should shutdown? I can sorta make this work, but not in a very clean >> >way. >> > >> >Just curious what others are doing about graceful shutdown. >> > >> >Thanks, >> >Zach >> >> >
