Thanks for the suggestion on stopasgroup. It looks like then I would have to run each gunicorn process on different ports. doesn't this require me to add more ports in nginx? That's minor still.
Except, if the whole parent children forking thing is true, then why is aurum able to kill without issue? http://pastebin.com/RtrSbxHX As the grep shows, Aurum is another project which runs with gunicorn_paster. They both have the same configuration except different virtualenv since they are different project. I can kill aurum without problem. as a side note: no graphyte isn't graphite. it's a project currently in development by CUNY City students. we do plan on releasing as open source in the future, however. it's a blackbox for collaborative computing. So I am trying to figure out what could be the issue given no code has changed. Thanks., John On Thu, Feb 21, 2013 at 4:17 PM, David Birdsong <[email protected]>wrote: > > > On Thu, Feb 21, 2013 at 1:15 PM, Chris McDonough <[email protected]> wrote: > >> On Thu, 2013-02-21 at 13:07 -0800, David Birdsong wrote: >> > >> > >> > On Thu, Feb 21, 2013 at 12:57 PM, John Wong <[email protected]> >> > wrote: >> > Hi Dan, >> > >> > >> > The fix didn't work. It still leave one child behind. >> > Like I said, this is my first time encountering the bug and we >> > didn't change any code at all. >> > Strangely, I have another guniconr_paster app running and it >> > can kill without problem. >> > >> > >> > >> > >> > supevisord only kills it's direct children, it doesn't go after any >> > grandchildren. >> > >> >> >> You can tell it to clean up processes left behind by ill-behaved >> children by using the "stopasgroup" option in the process config, I >> think. It was added pretty recently (in supervisor 3.0b1) >> > > oh, cool. thanks for the correction. i guess signaling a process group > makes sense. > > i'm stuck on a few revisions behind since my patch to include various > RLIMIT configurations is kinda crappy but we rely on it very heavily. > > >> - C >> >> > >> > Thanks. >> > >> > >> > John >> > >> > On Thu, Feb 21, 2013 at 1:36 PM, Dan Crosta <[email protected]> >> > wrote: >> > We use supervisor with gunicorn with stopsignal=INT >> > successfully -- might be worth a try. >> > >> > - Dan >> > >> > On Feb 21, 2013, at 1:09 PM, "David Birdsong" >> > <[email protected]> wrote: >> > >> > > if gunicorn is doing it's own forking, which i'm >> > guessing it does, >> > > then supervisord won't know about the child >> > processes. >> > > >> > > i've not used gunicorn, but when i run tornado >> > servers, i dont use >> > > tornado's process manager, i instead use >> > supervisord's argument >> > > templating and numprocs to fork direct children from >> > supervisord. the >> > > arg templating empowers you to pass in different >> > ports to each >> > > process. >> > > >> > > so instead you'd run two gunicorn processes directly >> > from supervisord >> > > on different ports. >> > > >> > > >> > > (aside, are you using gunicorn to run graphite >> > instead of django?) >> > > >> > > On Thu, Feb 21, 2013 at 9:54 AM, John Wong >> > <[email protected]> wrote: >> > >> Hi, >> > >> >> > >> I bumped into this biazzard problem today. I never >> > had any issue with >> > >> supervisord for the last two weeks since I started >> > using it. >> > >> >> > >> >> > >> > (gcs)giabadmin@giab-master:/opt/graphyte/gcs/tests/integration$ >> ps -elf|grep >> > >> "gcs" >> > >> 4 S 1011 3654 2899 1 80 0 - 3811 poll_s >> > 17:48 ? 00:00:01 >> > >> /opt/graphyte/vens/gcs/bin/python >> /opt/graphyte/vens/gcs/bin/gunicorn_paster >> > >> /opt/graphyte/gcs/development.ini -w 1 -t 3600 >> > >> 1 S 1011 3659 3654 8 80 0 - 11852 ep_pol >> > 17:48 ? 00:00:11 >> > >> /opt/graphyte/vens/gcs/bin/python >> /opt/graphyte/vens/gcs/bin/gunicorn_paster >> > >> /opt/graphyte/gcs/development.ini -w 1 -t 3600 >> > >> 1 S postgres 3674 1347 0 80 0 - 13192 sk_wai >> > 17:48 ? 00:00:00 >> > >> postgres: postgres gcs 127.0.0.1(47334) idle >> > >> 1 S postgres 3732 1347 0 80 0 - 13001 sk_wai >> > 17:49 ? 00:00:00 >> > >> postgres: postgres gcs 127.0.0.1(47419) idle >> > >> 0 R 1011 3747 1105 3 80 0 - 1157 - >> > 17:50 pts/0 00:00:00 >> > >> grep --color=auto gcs >> > >> >> > >> >> > >> We are looking at GCS. We have two of these. >> > >> >> > >> Then I ran super which is a shortcut for sudo >> > supervisrdctl (I made this >> > >> into a bash alias) >> > >> >> > >> >> > >> > (gcs)giabadmin@giab-master:/opt/graphyte/gcs/tests/integration$ >> super stop >> > >> gcs >> > >> gcs: stopped >> > >> >> > >> I waited maybe a minute and I saw >> > >> >> > >> >> > >> > (gcs)giabadmin@giab-master:/opt/graphyte/gcs/tests/integration$ >> ps -elf|grep >> > >> "gcs" >> > >> 1 S 1011 3659 1 4 80 0 - 11852 ep_pol >> > 17:48 ? 00:00:11 >> > >> /opt/graphyte/vens/gcs/bin/python >> /opt/graphyte/vens/gcs/bin/gunicorn_paster >> > >> /opt/graphyte/gcs/development.ini -w 1 -t 3600 >> > >> 1 S postgres 3674 1347 0 80 0 - 13192 sk_wai >> > 17:48 ? 00:00:00 >> > >> postgres: postgres gcs 127.0.0.1(47334) idle >> > >> 1 S postgres 3732 1347 0 80 0 - 13001 sk_wai >> > 17:49 ? 00:00:00 >> > >> postgres: postgres gcs 127.0.0.1(47419) idle >> > >> 0 R 1011 3784 1105 0 80 0 - 1157 - >> > 17:51 pts/0 00:00:00 >> > >> grep --color=auto gcs >> > >> >> > >> Odd. Still alive. 3659 is still alive. >> > >> >> > >> I can't start my app until I manually kills 3659. >> > Do we know what's causing >> > >> the trouble? >> > >> >> > >> This is my conf: >> > >> >> > >> [program:gcs] >> > >> command=/opt/graphyte/vens/gcs/bin/gunicorn_paster >> > >> /opt/graphyte/gcs/development.ini -w 1 -t 3600 >> > >> user=giabadmin >> > >> autostart=true >> > >> autorestart=false >> > >> stopsignal=QUIT >> > >> log_stdout=true >> > >> log_stderr=true >> > >> logfile=/var/log/graphyte/gcs2/supervisord.log >> > >> stdout_logfile=/var/log/graphyte/gcs2/stdout.log >> > >> stderr_logfile=/var/log/graphyte/gcs2/stderr.log >> > >> logfile_maxbytes=20MB >> > >> logfile_backups=10 >> > >> >> > >> Thanks and sorry for the long post. >> > >> >> > >> Cheers, >> > >> John >> > >> >> > >> _______________________________________________ >> > >> Supervisor-users mailing list >> > >> [email protected] >> > >> >> > >> https://lists.supervisord.org/mailman/listinfo/supervisor-users >> > > _______________________________________________ >> > > Supervisor-users mailing list >> > > [email protected] >> > > >> > >> https://lists.supervisord.org/mailman/listinfo/supervisor-users >> > _______________________________________________ >> > Supervisor-users mailing list >> > [email protected] >> > >> https://lists.supervisord.org/mailman/listinfo/supervisor-users >> > >> > >> > >> > >> > _______________________________________________ >> > Supervisor-users mailing list >> > [email protected] >> > https://lists.supervisord.org/mailman/listinfo/supervisor-users >> > >> > >> > _______________________________________________ >> > Supervisor-users mailing list >> > [email protected] >> > https://lists.supervisord.org/mailman/listinfo/supervisor-users >> >> >> >
_______________________________________________ Supervisor-users mailing list [email protected] https://lists.supervisord.org/mailman/listinfo/supervisor-users
