[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
mysql-5.7 is fixed anywhere we care; I'm marking the main task as Fix Released. ** Changed in: mysql-5.7 (Ubuntu) Status: Triaged => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
This bug was fixed in the package mysql-8.0 - 8.0.26-0ubuntu0.20.04.3 --- mysql-8.0 (8.0.26-0ubuntu0.20.04.3) focal; urgency=medium * d/systemd: Disable service timeout For large databases, the service could timeout on stop, possibly leading to data corruption during a system shutdown. (LP: #1882527) -- Paride Legovini Thu, 19 Aug 2021 15:32:33 +0200 ** Changed in: mysql-8.0 (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
This bug was fixed in the package mysql-5.7 - 5.7.35-0ubuntu0.18.04.2 --- mysql-5.7 (5.7.35-0ubuntu0.18.04.2) bionic; urgency=medium * d/systemd: Disable service timeout For large databases, the service could timeout on stop, possibly leading to data corruption during a system shutdown. (LP: #1882527) -- Paride Legovini Thu, 19 Aug 2021 18:34:52 +0200 ** Changed in: mysql-5.7 (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
The Bionic upload now has no issues preventing migration. The Focal upload is being held back by: https://bugs.launchpad.net/ubuntu/+source/ruby-mysql2/+bug/1942476 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
The reported Bionic regressions were false positives and have now been cleared. The Focal postfix regression was also a false positive, and it's not cleared. One regression remains on Focal: ruby-mysql2/0.5.2-1ubuntu3 which isn't really triggered by the mysql-8.0 as it's reproducible with what we already have in the archive, e.g. by doing: autopkgtest --no-built-binaries \ --setup-commands="sed -i '/^# deb-src/s/^# //' /etc/apt/sources.list" \ --setup-commands="apt-get update" \ ruby-mysql2 -- lxd ubuntu:focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
Verification done for mysql-8.0/focal again following the [Test Plan], so I tagged this overall verification-done. ** Tags removed: verification-needed verification-needed-focal ** Tags added: verification-done verification-done-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
Hello Warwick, or anyone else affected, Accepted mysql-8.0 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-8.0/8.0.26-0ubuntu0.20.04.3 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-focal. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: mysql-8.0 (Ubuntu Focal) Status: Triaged => Fix Committed ** Tags added: verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
I did the verification for mysql-5.7/bionic according to the [Test Plan], will proceed with the mysql-8.0/focal verification once it's accepted in -proposed. ** Tags removed: verification-needed-bionic ** Tags added: verification-done-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Changed in: mysql-5.7 (Ubuntu Bionic) Assignee: (unassigned) => Paride Legovini (paride) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
Focal is frozen for 20.04.3 right now, so I'll accept this into Bionic only into proposed for now. However we should release Focal before Bionic to avoid regressing Bionic users upgrading to Focal. ** Changed in: mysql-5.7 (Ubuntu Bionic) Status: New => Fix Committed ** Tags added: verification-needed verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Merge proposal linked: https://code.launchpad.net/~paride/ubuntu/+source/mysql-5.7/+git/mysql-5.7/+merge/407446 ** Merge proposal linked: https://code.launchpad.net/~paride/ubuntu/+source/mysql-8.0/+git/mysql-8.0/+merge/407447 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Description changed: [Impact] mysql-server-5.7.mysql.service (bionic) and mysql- server-8.0.mysql.service (focal) have a TimeoutSec=600. This has the effect of killing the MySQL process if this timeout is reached. Very large databases can exceed the 600s timeout, and a safe tradeoff between timing out at some point and waiting long enough to accommodate large/huge databases does not seem to exist. - This issue has been fixed in Debian and in Ubuntu >= Hirsute by - disabling the timeout (TimeoutSec=infinity). + This issue has been fixed in Debian and in Ubuntu >=Hirsute by disabling + the timeout (TimeoutSec=infinity). [Test Plan] - This is probably the most interesting bit of the SRU. + This is probably the most interesting bit of the SRU :-) In order to test this for real, as opposed to testing systemd's TimeoutSec, we need to make mysql very slow when it loads its tables. One way it to actually have huge tables, but I have no idea on how big they'd need to be. The other way is to slow down access to /var/lib/mysql at the I/O level. Something on these lines: apt install mysql-server-8.0 systemctl stop mysql cd /var/lib mv mysql mysql.bak truncate -s 300M mysql.blk losetup --show --find mysql.blk dmsetup create slowdev --table \ "0 $(blockdev --getsz /dev/loopX) delay /dev/loopX 0 100" # With /dev/loopX as printed by losetup, 100 = 100ms r/w delay # See: https://www.kernel.org/doc/Documentation/device-mapper/delay.txt mkfs.ext4 /dev/mapper/slowdev mkdir mysql mount /dev/mapper/slowdev mysql chown mysql:mysql mysql cp -av mysql.bak/* mysql time systemctl start mysql # slow! + systemctl status mysql By tuning the delay parameter it is possible to trigger the timeout with the pre-SRU package, and verify that post-SRU it can load in more than 10 minutes. Note: this can be tested in an LXD VM but it requires booting the linux-image-generic kernel, as linux-image-kvm doesn't ship the dm delay target. The lxd-agent won't work, just ssh-import-id and ssh in. (I think this is overkill for this SRU, but it's a quite general way to make stuff slow. I'm sure it will come useful in other cases!) [Where problems could occur] The TimeoutSec=infinity syntax is supported by the systemd versions in all the supported releases of Ubuntu, so this won't be a problem. Then only change in behavior due to this change will happen on systems where the timeout is reached, and mysql is thus killed. In these cases the database server wouldn't be running in any case, but there could be cases of bad or overgrown databases (e.g. because of a runaway script adding infinite data) where the timeout is doing the right thing, - preventing mysql from consuming system resources. In these already - broken systems TimeoutSec=infinity may increase the breakage. This won't - affect working production systems. + preventing mysql from consuming system resources forever. In these + already broken systems TimeoutSec=infinity may increase the breakage. + This won't affect working production systems. [Development Fix] [Stable Fix] The same fix is already already landed in Hirsute, Impish and Debian unstable. [Original Description] MySQL on 20.04 has TimeoutSec set to 600 (IIRC) in the systemd script. This has the effect of killing the MySQL process if this timeout is reached. IMHO this is a Very Bad Idea. A database server process should only be force killed by a user action. I would prefer that the server had unlimited time to cleanly shutdown and startup (eg if recovering). Our DB is about 500GB with some very large tables (for us at least) eg. 250GB and we've had more than a few unfortunate delays as a result of delayed startup caused by recoveries because MySQL was killed prematurely. Because MySQL 8.0 has reduced the default logging level, it was not clear to me that the process was being force killed. I believe the MySQL team are of the same view as me per https://bugs.mysql.com/bug.php?id=91423: ``` [12 Jul 2019 15:57] Paul Dubois Posted by developer: Fixed in 8.0.18. On Debian, long InnoDB recovery times at startup could cause systemd service startup failure. The default systemd service timeout is now disabled (consistent with RHEL) to prevent this from happening. ``` -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Description changed: [Impact] mysql-server-5.7.mysql.service (bionic) and mysql- server-8.0.mysql.service (focal) have a TimeoutSec=600. This has the effect of killing the MySQL process if this timeout is reached. Very large databases can exceed the 600s timeout, and a safe tradeoff between timing out at some point and waiting long enough to accommodate large/huge databases does not seem to exist. This issue has been fixed in Debian and in Ubuntu >= Hirsute by disabling the timeout (TimeoutSec=infinity). [Test Plan] This is probably the most interesting bit of the SRU. In order to test this for real, as opposed to testing systemd's TimeoutSec, we need to make mysql very slow when it loads its tables. One way it to actually have huge tables, but I have no idea on how big they'd need to be. The other way is to slow down access to /var/lib/mysql at the I/O level. Something on these lines: - apt install mysql-server-8.0 - systemctl stop mysql - cd /var/lib - mv mysql mysql.bak - truncate -s 300M mysql.blk - losetup --show --find mysql.blk + apt install mysql-server-8.0 + systemctl stop mysql + cd /var/lib + mv mysql mysql.bak + truncate -s 300M mysql.blk + losetup --show --find mysql.blk - dmsetup create slowdev --table \ - "0 $(blockdev --getsz /dev/loopX) delay /dev/loopX 0 100" - # With /dev/loopX as printed by losetup, 100 = 100ms r/w delay - # See: https://www.kernel.org/doc/Documentation/device-mapper/delay.txt + dmsetup create slowdev --table \ + "0 $(blockdev --getsz /dev/loopX) delay /dev/loopX 0 100" + # With /dev/loopX as printed by losetup, 100 = 100ms r/w delay + # See: https://www.kernel.org/doc/Documentation/device-mapper/delay.txt - mkfs.ext4 /dev/mapper/slowdev - mkdir mysql - mount /dev/mapper/slowdev mysql - chown mysql:mysql mysql - cp -av mysql.bak/* mysql + mkfs.ext4 /dev/mapper/slowdev + mkdir mysql + mount /dev/mapper/slowdev mysql + chown mysql:mysql mysql + cp -av mysql.bak/* mysql - systemctl start mysql # slow! + time systemctl start mysql # slow! By tuning the delay parameter it is possible to trigger the timeout with the pre-SRU package, and verify that post-SRU it can load in more than 10 minutes. Note: this can be tested in an LXD VM but it requires booting the linux-image-generic kernel, as linux-image-kvm doesn't ship the dm delay target. The lxd-agent won't work, just ssh-import-id and ssh in. (I think this is overkill for this SRU, but it's a quite general way to make stuff slow. I'm sure it will come useful in other cases!) [Where problems could occur] The TimeoutSec=infinity syntax is supported by the systemd versions in all the supported releases of Ubuntu, so this won't be a problem. Then only change in behavior due to this change will happen on systems where the timeout is reached, and mysql is thus killed. In these cases the database server wouldn't be running in any case, but there could be cases of bad or overgrown databases (e.g. because of a runaway script adding infinite data) where the timeout is doing the right thing, preventing mysql from consuming system resources. In these already broken systems TimeoutSec=infinity may increase the breakage. This won't affect working production systems. [Development Fix] [Stable Fix] The same fix is already already landed in Hirsute, Impish and Debian unstable. [Original Description] MySQL on 20.04 has TimeoutSec set to 600 (IIRC) in the systemd script. This has the effect of killing the MySQL process if this timeout is reached. IMHO this is a Very Bad Idea. A database server process should only be force killed by a user action. I would prefer that the server had unlimited time to cleanly shutdown and startup (eg if recovering). Our DB is about 500GB with some very large tables (for us at least) eg. 250GB and we've had more than a few unfortunate delays as a result of delayed startup caused by recoveries because MySQL was killed prematurely. Because MySQL 8.0 has reduced the default logging level, it was not clear to me that the process was being force killed. I believe the MySQL team are of the same view as me per https://bugs.mysql.com/bug.php?id=91423: ``` [12 Jul 2019 15:57] Paul Dubois Posted by developer: Fixed in 8.0.18. On Debian, long InnoDB recovery times at startup could cause systemd service startup failure. The default systemd service timeout is now disabled (consistent with RHEL) to prevent this from happening. ``` ** Also affects: mysql-5.7 (Ubuntu) Importance: Undecided Status: New ** No longer affects: mysql-5.7 (Ubuntu Eoan) ** No longer affects: mysql-5.7 (Ubuntu Focal) ** No longer affects: mysql-5.7 (Ubuntu Bionic) ** Changed in: mysql-5.7 (Ubuntu) Status: New => Triaged ** No longer affects: mysql-8.0 (Ub
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Description changed: [Impact] mysql-server-5.7.mysql.service (bionic) and mysql- server-8.0.mysql.service (focal) have a TimeoutSec=600. This has the effect of killing the MySQL process if this timeout is reached. Very large databases can exceed the 600s timeout, and a safe tradeoff between timing out at some point and waiting long enough to accommodate large/huge databases does not seem to exist. This issue has been fixed in Debian and in Ubuntu >= Hirsute by disabling the timeout (TimeoutSec=infinity). [Test Plan] + + This is probably the most interesting bit of the SRU. + + In order to test this for real, as opposed to testing systemd's + TimeoutSec, we need to make mysql very slow when it loads its tables. + One way it to actually have huge tables, but I have no idea on how big + they'd need to be. The other way is to slow down access to + /var/lib/mysql at the I/O level. Something on these lines: + + apt install mysql-server-8.0 + systemctl stop mysql + cd /var/lib + mv mysql mysql.bak + truncate -s 300M mysql.blk + losetup --show --find mysql.blk + + dmsetup create slowdev --table \ + "0 $(blockdev --getsz /dev/loopX) delay /dev/loopX 0 100" + # With /dev/loopX as printed by losetup, 100 = 100ms r/w delay + # See: https://www.kernel.org/doc/Documentation/device-mapper/delay.txt + + mkfs.ext4 /dev/mapper/slowdev + mkdir mysql + mount /dev/mapper/slowdev mysql + chown mysql:mysql mysql + cp -av mysql.bak/* mysql + + systemctl start mysql # slow! + + By tuning the delay parameter it is possible to trigger the timeout with + the pre-SRU package, and verify that post-SRU it can load in more than + 10 minutes. + + Note: this can be tested in an LXD VM but it requires booting the linux-image-generic kernel, as linux-image-kvm doesn't ship the dm delay target. + The lxd-agent won't work, just ssh-import-id and ssh in. + + (I think this is overkill for this SRU, but it's a quite general way to + make stuff slow. I'm sure it will come useful in other cases!) [Where problems could occur] The TimeoutSec=infinity syntax is supported by the systemd versions in all the supported releases of Ubuntu, so this won't be a problem. Then only change in behavior due to this change will happen on systems where the timeout is reached, and mysql is thus killed. In these cases the database server wouldn't be running in any case, but there could be cases of bad or overgrown databases (e.g. because of a runaway script adding infinite data) where the timeout is doing the right thing, preventing mysql from consuming system resources. In these already broken systems TimeoutSec=infinity may increase the breakage. This won't affect working production systems. [Development Fix] [Stable Fix] The same fix is already already landed in Hirsute, Impish and Debian unstable. [Original Description] MySQL on 20.04 has TimeoutSec set to 600 (IIRC) in the systemd script. This has the effect of killing the MySQL process if this timeout is reached. IMHO this is a Very Bad Idea. A database server process should only be force killed by a user action. I would prefer that the server had unlimited time to cleanly shutdown and startup (eg if recovering). Our DB is about 500GB with some very large tables (for us at least) eg. 250GB and we've had more than a few unfortunate delays as a result of delayed startup caused by recoveries because MySQL was killed prematurely. Because MySQL 8.0 has reduced the default logging level, it was not clear to me that the process was being force killed. I believe the MySQL team are of the same view as me per https://bugs.mysql.com/bug.php?id=91423: ``` [12 Jul 2019 15:57] Paul Dubois Posted by developer: Fixed in 8.0.18. On Debian, long InnoDB recovery times at startup could cause systemd service startup failure. The default systemd service timeout is now disabled (consistent with RHEL) to prevent this from happening. ``` -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Description changed: [Impact] mysql-server-5.7.mysql.service (bionic) and mysql- server-8.0.mysql.service (focal) have a TimeoutSec=600. This has the effect of killing the MySQL process if this timeout is reached. Very large databases can exceed the 600s timeout, and a safe tradeoff between timing out at some point and waiting long enough to accommodate large/huge databases does not seem to exist. This issue has been fixed in Debian and in Ubuntu >= Hirsute by disabling the timeout (TimeoutSec=infinity). - [Test Case] + [Test Plan] [Where problems could occur] + The TimeoutSec=infinity syntax is supported by the systemd versions in + all the supported releases of Ubuntu, so this won't be a problem. + + Then only change in behavior due to this change will happen on systems + where the timeout is reached, and mysql is thus killed. In these cases + the database server wouldn't be running in any case, but there could be + cases of bad or overgrown databases (e.g. because of a runaway script + adding infinite data) where the timeout is doing the right thing, + preventing mysql from consuming system resources. In these already + broken systems TimeoutSec=infinity may increase the breakage. This won't + affect working production systems. + [Development Fix] + [Stable Fix] - [Stable Fix] + The same fix is already already landed in Hirsute, Impish and Debian + unstable. [Original Description] MySQL on 20.04 has TimeoutSec set to 600 (IIRC) in the systemd script. This has the effect of killing the MySQL process if this timeout is reached. IMHO this is a Very Bad Idea. A database server process should only be force killed by a user action. I would prefer that the server had unlimited time to cleanly shutdown and startup (eg if recovering). Our DB is about 500GB with some very large tables (for us at least) eg. 250GB and we've had more than a few unfortunate delays as a result of delayed startup caused by recoveries because MySQL was killed prematurely. Because MySQL 8.0 has reduced the default logging level, it was not clear to me that the process was being force killed. I believe the MySQL team are of the same view as me per https://bugs.mysql.com/bug.php?id=91423: ``` [12 Jul 2019 15:57] Paul Dubois Posted by developer: Fixed in 8.0.18. On Debian, long InnoDB recovery times at startup could cause systemd service startup failure. The default systemd service timeout is now disabled (consistent with RHEL) to prevent this from happening. ``` -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Description changed: + [Impact] + + mysql-server-5.7.mysql.service (bionic) and mysql- + server-8.0.mysql.service (focal) have a TimeoutSec=600. This has the + effect of killing the MySQL process if this timeout is reached. + + Very large databases can exceed the 600s timeout, and a safe tradeoff + between timing out at some point and waiting long enough to accommodate + large/huge databases does not seem to exist. + + This issue has been fixed in Debian and in Ubuntu >= Hirsute by + disabling the timeout (TimeoutSec=infinity). + + [Test Case] + + [Where problems could occur] + + [Development Fix] + + [Stable Fix] + + [Original Description] + MySQL on 20.04 has TimeoutSec set to 600 (IIRC) in the systemd script. This has the effect of killing the MySQL process if this timeout is reached. IMHO this is a Very Bad Idea. A database server process should only be force killed by a user action. I would prefer that the server had unlimited time to cleanly shutdown and startup (eg if recovering). Our DB is about 500GB with some very large tables (for us at least) eg. 250GB and we've had more than a few unfortunate delays as a result of delayed startup caused by recoveries because MySQL was killed prematurely. Because MySQL 8.0 has reduced the default logging level, it was not clear to me that the process was being force killed. I believe the MySQL team are of the same view as me per https://bugs.mysql.com/bug.php?id=91423: ``` [12 Jul 2019 15:57] Paul Dubois Posted by developer: Fixed in 8.0.18. On Debian, long InnoDB recovery times at startup could cause systemd service startup failure. The default systemd service timeout is now disabled (consistent with RHEL) to prevent this from happening. ``` -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Changed in: mysql-8.0 (Ubuntu Bionic) Assignee: (unassigned) => Paride Legovini (paride) ** Changed in: mysql-8.0 (Ubuntu Focal) Assignee: (unassigned) => Paride Legovini (paride) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
> This is fixed in Hirsute now. Backports to Focal and Bionic may > be appropriate. Agreed. I'll ask Miriam or Paride to TAL. Thanks! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
This bug was fixed in the package mysql-8.0 - 8.0.23-1ubuntu2 --- mysql-8.0 (8.0.23-1ubuntu2) hirsute; urgency=medium * Add missing libzstd-dev on runtime dev package (Closes: #981905) -- Gianfranco Costamagna Thu, 04 Feb 2021 22:30:12 +0100 ** Changed in: mysql-8.0 (Ubuntu) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
This is fixed in Hirsute now. Backports to Focal and Bionic may be appropriate. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Description changed: MySQL on 20.04 has TimeoutSec set to 600 (IIRC) in the systemd script. This has the effect of killing the MySQL process if this timeout is reached. IMHO this is a Very Bad Idea. A database server process should only be force killed by a user action. I would prefer that the server had unlimited time to cleanly shutdown and startup (eg if recovering). Our DB is about 500GB with some very large tables (for us at least) eg. 250GB and we've had more than a few unfortunate delays as a result of delayed startup caused by recoveries because MySQL was killed prematurely. Because MySQL 8.0 has reduced the default logging level, it was not clear to me that the process was being force killed. I believe the MySQL team are of the same view as me per https://bugs.mysql.com/bug.php?id=91423: ``` [12 Jul 2019 15:57] Paul Dubois Posted by developer: - + Fixed in 8.0.18. On Debian, long InnoDB recovery times at startup could cause systemd service startup failure. The default systemd service timeout is now disabled (consistent with RHEL) to prevent this from happening. ``` -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
I'm unassigning myself as I'm not sure what the next steps are but I believe it should be corrected by ubuntu taking the latest debian version of the package (i.e. mysql-8.0 (8.0.22-1)) -> no ubuntu-patch needed ** Changed in: mysql-8.0 (Ubuntu) Assignee: Koen (koen-beek) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
hmm when doing a git clone https://salsa.debian.org/mariadb-team/mysql.git -b mysql-8.0/debian/master it seems it has been fixed by putting infinity to TimeoutSec (I suppose that's the same as 0) in Debian mysql/debian/mysql-server-8.0.mysql.service:TimeoutSec=infinity -> seems to have been fixed in Debian in mysql-8.0.0.22 and https://metadata.ftp- master.debian.org/changelogs//main/m/mysql-8.0/mysql-8.0_8.0.22-1_changelog shows : mysql-8.0 (8.0.22-1) unstable; urgency=medium ... * d/systemd: Disable service timeout For large databases, the service could timeout on stop, possibly leading to data corruption during a system shutdown. (LP: #1882527) --> it's in debian in the next patch release Ubuntu is currently on 8.0.22-0(ubuntu0.20.10.2) -> ubuntu just needs to align with debian 8.0.22-1 instead of 8.0.22-0 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
There does not seem to be an equivalent debian bug about this issue yet will make a patch in ubuntu and forward it to debian : https://wiki.ubuntu.com/Debian/Bugs -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
The mysql bug ticket (https://bugs.mysql.com/bug.php?id=91423) mentions [12 Jul 2019 15:57] Paul Dubois Posted by developer: Fixed in 8.0.18. On Debian, long InnoDB recovery times at startup could cause systemd service startup failure. The default systemd service timeout is now disabled (consistent with RHEL) to prevent this from happening. -- the latest ubuntu code (based on 8.0.22) contains following mentions of TimeoutSec koen@hpubuntu:~/Dev/personal/ubuntu/mysql$ grep -rR TimeoutSec * mysql-8.0-8.0.22/packaging/deb-in/mysql-packagesource-server.mysql.service.in:TimeoutSec=0 mysql-8.0-8.0.22/packaging/deb-in/mysql-packagesource-server.my...@.service.in:TimeoutSec=0 mysql-8.0-8.0.22/scripts/systemd/mys...@.service.in:TimeoutSec=0 mysql-8.0-8.0.22/scripts/systemd/mysqld.service.in:TimeoutSec=0 mysql-8.0-8.0.22/debian/mysql-server-8.0.mysql.service:TimeoutSec=600 --> it seems there's still 1 mention of 600 seconds in the Debian/ubuntu package I can take on this issue but it might take a few weeks forme to complete - ETA mid Jan 2021 as I'm pretty new at ubuntu bugfixing/packaging and I'll have some questions for some more expert people - correct patch description - how to involve Debian upstream - how to make patches for several releases (I'll start with hirsute) if it's more urgent don't hesitate to assign someone else ** Bug watch added: MySQL Bug System #91423 http://bugs.mysql.com/bug.php?id=91423 ** Changed in: mysql-8.0 (Ubuntu) Assignee: (unassigned) => Koen (koen-beek) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
The Eoan Ermine has reached end of life, so this bug will not be fixed for that release ** Changed in: mysql-8.0 (Ubuntu Eoan) Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Tags added: bitesize -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Also affects: mysql-8.0 (Ubuntu Eoan) Importance: Undecided Status: New ** Also affects: mysql-8.0 (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: mysql-8.0 (Ubuntu Groovy) Importance: Undecided Status: Confirmed ** Also affects: mysql-8.0 (Ubuntu Focal) Importance: Undecided Status: New ** No longer affects: mysql-8.0 (Ubuntu Groovy) ** Changed in: mysql-8.0 (Ubuntu Focal) Status: New => Triaged ** Changed in: mysql-8.0 (Ubuntu Eoan) Status: New => Triaged ** Changed in: mysql-8.0 (Ubuntu Bionic) Status: New => Triaged ** Changed in: mysql-8.0 (Ubuntu Bionic) Importance: Undecided => Medium ** Changed in: mysql-8.0 (Ubuntu Eoan) Importance: Undecided => Medium ** Changed in: mysql-8.0 (Ubuntu Focal) Importance: Undecided => Medium ** Changed in: mysql-8.0 (Ubuntu) Importance: Undecided => High ** Tags added: server-next -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
The vendor is working on it - we can assist to that. ** Changed in: mysql-8.0 (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
Thanks Daniel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
** Package changed: xorg-server (Ubuntu) => mysql-8.0 (Ubuntu) ** Tags added: focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1882527] Re: mysql timeoutsec results in killing mysql process
I see I have created this bug report in xorg instead of MySQL. Please accept my apology and either move or delete. I am happy to recreate, just tell me which. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1882527 Title: mysql timeoutsec results in killing mysql process To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1882527/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
