Hey Zach, It sounds likely that the BrokerProxy thread is blocking or improperly catching an interrupt exception.
Can you take a thread dump? My guess is that you'll see that the main thread is blocked on BrokerProxy.stop: thread.interrupt thread.join It'll likely be blocked on thread.join. If that's the case, I'd like to see what the BrokerProxy thread is doing. This line indicates that the thread seems to be shutting down: [DEBUG] [2015-01-14 17:19:12,596] [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:9192 for client samza_consumer-twitter_message_separation-1-1421255914862-1] o.a.s.s.k.DefaultFetchSimpleConsumer: Disconnecting from localhost:9192 But if thread.join is blocking, then the shutdown never seems to complete. It'd be good to see where these threads are at. Cheers, Chris On 1/14/15 10:49 AM, "Zach Cox" <[email protected]> wrote: >Hi - related to the discussion yesterday about graceful shutdown [1], >today >I can't seem to get the SamzaContainer to actually shut down. Yesterday I >was seeing nice, fast, complete shutdown logs [2]. However, today the last >log line I see is related to shutting down BrokerProxy, then nothing else >happens until the container is finally just SIGKILLed by Docker [3]. > >The consistency with which clean, fast shutdowns were happening yesterday >and now they are never happening today leads me to believe I've changed >something, but I really can't find anything that would lead to this. It's >almost like there's some deadlock in BrokerProxy, but I would think that >would show up randomly - I've tried this about 100 times today, and every >time the logs are as shown in [3]. Today I also see 2 exceptions in the >Kafka broker logs [4] but I don't know if those were occurring yesterday >when Samza was cleanly shutting down, or not. > >One thing that doesn't seem to be happening today is this log line from >BrokerProxy: "Got interrupt exception in broker proxy thread." > >Has anyone seen this, or have any advice on what to try next? > >Thanks, >Zach > > >[1] >http://www.mail-archive.com/[email protected]/msg02246.html > >[2] >https://gist.githubusercontent.com/zcox/6ec8910bd3f18e36c1a2/raw/666eae245 >11490bf51a66e56fd90c794ea6b6282/stdout > >[3] >https://gist.githubusercontent.com/zcox/673a2ba607c566de7650/raw/d29ea8b99 >868da9cc4dc1db198315ee1d03bc694/gistfile1.txt > >[4] >https://gist.githubusercontent.com/zcox/f08de55c5d5fe2d70cde/raw/c4be453b5 >f6a57cfb5acb08508667f4b8ce8c2bd/gistfile1.txt
