[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2012-02-15 Thread Launchpad Bug Tracker
This bug was fixed in the package mysql-5.5 - 5.5.20-0ubuntu1 --- mysql-5.5 (5.5.20-0ubuntu1) precise; urgency=low * New upstream release. * d/mysql-server-5.5.mysql.upstart: Fix stop on to make sure mysql is fully stopped before shutdown commences. (LP: #688541) Also

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2012-01-02 Thread Launchpad Bug Tracker
This bug was fixed in the package sysvinit - 2.88dsf-13.10ubuntu4.1 --- sysvinit (2.88dsf-13.10ubuntu4.1) oneiric-proposed; urgency=low * d/src/initscripts/etc/init.d/sendsigs: wait up to 300 extra seconds for upstart jobs that have been killed. They will be sent SIGKILL by

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-12-16 Thread Alan
Tested the Python test case from the bug description using the initscripts in Oneiric proposed. Prior to patch, did see Killing all remaining processes... fail as described. After patch, saw All processes ended within 16 seconds. According to the test case, this is a successful fix. -- You

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-12-16 Thread Clint Byrum
Alan you are my stable release update hero this week. :) The update should land in about 5 days (minimum 7 in -proposed just in case we missed a major regression.) ** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-12-14 Thread Martin Pitt
Hello Michael, or anyone else affected, Accepted sysvinit into oneiric-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-12-12 Thread Launchpad Bug Tracker
This bug was fixed in the package sysvinit - 2.88dsf-13.10ubuntu8 --- sysvinit (2.88dsf-13.10ubuntu8) precise; urgency=low * d/src/initscripts/etc/init.d/sendsigs: wait up to 300 extra seconds for upstart jobs that have been killed. They will be sent SIGKILL by upstart when

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-12-12 Thread Clint Byrum
** Also affects: mysql-5.1 (Ubuntu Oneiric) Importance: Undecided Status: New ** Also affects: mysql-5.5 (Ubuntu Oneiric) Importance: Undecided Status: New ** Also affects: sysvinit (Ubuntu Oneiric) Importance: Undecided Status: New ** Changed in: mysql-5.1 (Ubuntu

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-12-12 Thread Clint Byrum
Fix is waiting in the oneiric-proposed queue. ** Description changed: + == SRU JUSTIFICATION == + + IMPACT: potential data loss or extension of downtime. MySQL, for + example, if sent a SIGKILL before it is done flushing its buffers into + MyISAM tables, will lose that data. If using InnoDB,

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-12-09 Thread Clint Byrum
** Changed in: mysql-5.5 (Ubuntu Precise) Status: In Progress = Invalid ** Changed in: mysql-5.1 (Ubuntu Precise) Status: Triaged = Invalid ** Changed in: sysvinit (Ubuntu Precise) Status: Triaged = In Progress ** Changed in: sysvinit (Ubuntu Precise) Assignee:

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-12-09 Thread Launchpad Bug Tracker
** Branch linked: lp:~clint-fewbar/ubuntu/precise/sysvinit/wait-for- long-shutdown-jobs -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to mysql-5.1 in Ubuntu. https://bugs.launchpad.net/bugs/688541 Title: race condition on shutdown

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-11-30 Thread Clint Byrum
** Changed in: sysvinit (Ubuntu) Importance: Undecided = High ** Changed in: sysvinit (Ubuntu) Assignee: (unassigned) = Canonical Foundations Team (canonical-foundations) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-11-30 Thread Clint Byrum
So, I believe the right way to handle this is to wait a long time for any upstart job that has status of 'stop/killed'. We can't be finding all of the services that are slow to shutdown one by one. Authors of upstart jobs will know how long to wait before sending kill -9. Once kill -9 has been

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-11-30 Thread Clint Byrum
** Also affects: mysql-5.5 (Ubuntu) Importance: Undecided Status: New ** Changed in: mysql-5.5 (Ubuntu) Status: New = In Progress ** Changed in: mysql-5.5 (Ubuntu) Importance: Undecided = High -- You received this bug notification because you are a member of Ubuntu Server

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-11-30 Thread Clint Byrum
** Changed in: mysql-5.5 (Ubuntu) Assignee: (unassigned) = Clint Byrum (clint-fewbar) ** Changed in: mysql-5.1 (Ubuntu) Status: In Progress = Triaged -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to mysql-5.1 in Ubuntu.

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-11-30 Thread Steve Langasek
I agree that this should be fixed in sysvinit. Is it really appropriate to change mysql-5.5 at all? It should be a straightforward change to sysvinit, and the mysql change should be reverted afterwards. ** Tags added: rls-p-tracking ** Also affects: mysql-5.1 (Ubuntu Precise) Importance:

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2011-11-30 Thread Clint Byrum
I think ultimately no, mysql-5.5 shouldn't be changed for this, and agreed the change should be straight forward. I was not certain if we were willing to tackle the sysvinit change in precise.. but on second thought, of course we should be. So I'll hold back on the change to mysql-5.5 and take a

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-28 Thread ingo
I do unsubscribe from this bug for the time beeing. It does not make sense to deal with the symtoms until the root of the evil Bug #672177 is fixed. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to mysql-5.1 in ubuntu.

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-24 Thread ingo
I first tried to grep within a x-term, but there I get faulty output regarding gvfs. I suppose it's not needed. So I booted into a maintainence root-shell (without network) and did: lsof -n | grep deleted - nothing reported apt-get install --reinstall libc6 and afterwards lsof -n | grep

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-24 Thread ingo
And here the output of initctl list after reinstall of libc6. It is amaizeing, that even though I selected root shell without network in the maintainence system, a lot of services including network is up and running (I used scp to copy the output to my PC). ** Attachment added: initctl list

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-23 Thread ingo
@Clint: I did test your proposal in Maverick. Before editing the stop scripts: fgrep stop on runlevel /etc/init/*.conf /etc/init/acpid.conf:stop on runlevel [!2345] /etc/init/anacron.conf:stop on runlevel [!2345] /etc/init/apport.conf:stop on runlevel [!2345] /etc/init/atd.conf:stop on runlevel

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-23 Thread ingo
I took you literally and canged all [!2345], not the others: The remaining now are: fgrep stop on runlevel /etc/init/*.conf /etc/init/rc.conf:stop on runlevel [!$RUNLEVEL] /etc/init/rcS.conf:stop on runlevel [!S] /etc/init/rc-sysinit.conf:stop on runlevel /etc/init/tty2.conf:stop on runlevel

Re: [Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-23 Thread Clint Byrum
On Thu, 2010-12-23 at 22:07 +, ingo wrote: I took you literally and canged all [!2345], not the others: The remaining now are: fgrep stop on runlevel /etc/init/*.conf /etc/init/rc.conf:stop on runlevel [!$RUNLEVEL] /etc/init/rcS.conf:stop on runlevel [!S]

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-22 Thread ingo
This doesn't affect Debian as the upstart package in Debian still uses plain sysv compat and there are no native upstart jobs yet. A wise decision, good to know. Thanks, Ingo -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to mysql-5.1

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-22 Thread Clint Byrum
I was working with someone on another issue recently, and he pointed out a situation where someone had used this: start on starting rc RUNLEVEL=[06] to run a specific task before the system shut down. It got me thinking, should we instead just transition services that need to start before

Re: [Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-21 Thread Clint Byrum
On Tue, 2010-12-21 at 12:41 +, Scott James Remnant wrote: On 20/12/10 18:22, Clint Byrum wrote: In a message to ubuntu-devel I suggested that we have an abstract job, 'network-services', which most normal (non boot-critical) services should follow.

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-21 Thread James Hunt
@Michael: yes, this should be stop on unmount-filesystem or single- user (we can create a new event for single-user to make the logic clearer). @Clint: I agree that full migration sounds like the best approach. I have had a few discussions previously with Scott on the idea of abstract jobs. There

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-21 Thread ingo
On Tue, 2010-12-21 at 12:41 +, Scott James Remnant wrote: General note: ubuntu-devel is *NOT* the correct list to discuss Upstart changes unless they're unique to Ubuntu. Wouldn't it be fair to inform Debian about those problems before they release Squeeze? (tough I never observed it on

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-20 Thread James Hunt
After discussion with Scott, the best short-term solution would seem to be: 1) Modify /etc/init.d/umountfs to call the following in do_stop before calling umount/swapoff: initctl emit unmount-filesystem 2) Modify /etc/init.d/umountroot to call the following in do_stop before calling

Re: [Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-20 Thread Michael Biebl
2010/12/20 James Hunt 688...@bugs.launchpad.net: 3) Modify all upstart configs for services which are slow to stop such that they stop on unmount-filesystem,    rather than stop on runlevel [016]. - What about single user mode? I guess when switching to runlevel 1 we want to stop services

Re: [Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-20 Thread Clint Byrum
On Mon, 2010-12-20 at 12:50 +, James Hunt wrote: After discussion with Scott, the best short-term solution would seem to be: 1) Modify /etc/init.d/umountfs to call the following in do_stop before calling umount/swapoff: initctl emit unmount-filesystem 2) Modify

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-16 Thread Clint Byrum
So I've done some more thinking about this, and I had a bit of an aha! moment. While we *should* in fact stop using 'stop on runlevel [016]' or 'stop on runlevel [!2345]', I think we can solve this without touching all of those jobs. /etc/init.d/sendsigs has this code: # Upstart jobs

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-16 Thread Clint Byrum
** Patch added: lp688541.diff https://bugs.launchpad.net/ubuntu/+source/mysql-5.1/+bug/688541/+attachment/1767453/+files/lp688541.diff ** Changed in: sysvinit (Ubuntu) Status: New = Triaged -- You received this bug notification because you are a member of Ubuntu Server Team, which

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-16 Thread Clint Byrum
Also attaching the bash script that I used to test this, which simulates a process taking a long time on SIGTERM without forking.. it *should* work with sleep too, given the sendsigs change I posted, but when that change is not there.. sendsigs kills the sleeps and ruins all the fun. Below is the

Re: [Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-16 Thread Michael Biebl
2010/12/16 Clint Byrum cl...@fewbar.com: /etc/init.d/sendsigs has this code:        # Upstart jobs have their own stop on clauses that sends        # SIGTERM/SIGKILL just like this, so if they're still running,        # they're supposed to be        for pid in $(initctl list | sed -n -e

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-16 Thread Brian Murray
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to mysql-5.1 in ubuntu. https://bugs.launchpad.net/bugs/688541 Title: race condition on shutdown (leads to corrupted fs) -- Ubuntu-server-bugs mailing list

Re: [Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-16 Thread Clint Byrum
On Thu, 2010-12-16 at 15:45 +, Michael Biebl wrote: 2010/12/16 Clint Byrum cl...@fewbar.com: I'm attaching a debdiff that solves the race as far as I can tell, though I think it needs a good long look, since it could mean shutdowns hang for a long time waiting (I'm especially

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-15 Thread Clint Byrum
Hmm, I am wondering now if this bug is the same thing. https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/616287 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to mysql-5.1 in ubuntu. https://bugs.launchpad.net/bugs/688541 Title:

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-13 Thread Clint Byrum
Note that there is one incorrect assumption, which is that sendsigs will never kill any upstart jobs. In fact, it does make one attempt to kill -9 any still running upstart jobs: if [ -z $alldead ] ; then log_action_begin_msg Killing all remaining processes

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-13 Thread Clint Byrum
And, whoops, I just re-read that, its using killall5's -o to still omit those processes. Please disregard that last message then. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to mysql-5.1 in ubuntu.

Re: [Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-13 Thread Michael Biebl
2010/12/14 Clint Byrum cl...@fewbar.com: I do think the appropriate fix is to have umountfs emit an 'unmounting- filesystems' event and anything that does a 'start on local-filesystems' or 'start on filesystem' should also 'stop on unmounting-filesystems', What do you do about services which

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-13 Thread Clint Byrum
Great point, thanks for pointing that out! rc-sysinit does not start until filesystem and net-device-up IFACE=lo, and so, runlevel 2, which is reached by callint rc-sysinit, implies all of the services you mention. It is important to point out that we must include any of those *implied* to be

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-10 Thread Michael Biebl
-- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to mysql-5.1 in ubuntu. https://bugs.launchpad.net/bugs/688541 Title: race condition on shutdown (leads to corrupted fs) -- Ubuntu-server-bugs mailing list

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-10 Thread Martin Pitt
What would be the general approach to express shut down on runlevel 0/1/6 before the disks go away in terms of upstart triggers? Once there's an approach, pleaes hand over to canonical-server. Thanks! ** Changed in: mysql-5.1 (Ubuntu) Status: New = Triaged ** Changed in: mysql-5.1

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-10 Thread Ante Karamatić
Suggestion: make umountfs wait for all upstart jobs to finish. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to mysql-5.1 in ubuntu. https://bugs.launchpad.net/bugs/688541 Title: race condition on shutdown (leads to corrupted fs) --

Re: [Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-10 Thread Michael Biebl
2010/12/10 Ante Karamatić iv...@grad.hr: Suggestion: make umountfs wait for all upstart jobs to finish. Doesn't that conflict though with what is written in /etc/init.d/sendsigs: # Upstart jobs have their own stop on clauses that sends # SIGTERM/SIGKILL just like this, so if

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-10 Thread Clint Byrum
Ante, good eyes there. That statement is a little misleading, that if they're still running, they're supposed to be, as this assumes there was an event somewhere between the running system, and runlevel [016], which to my knowledge, there isn't. I'm a little confused as to why umountfs is still

[Bug 688541] Re: race condition on shutdown (leads to corrupted fs)

2010-12-10 Thread Robbie Williamson
James, could you take a look at this? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to mysql-5.1 in ubuntu. https://bugs.launchpad.net/bugs/688541 Title: race condition on shutdown (leads to corrupted fs) -- Ubuntu-server-bugs