Re: [ceph-users] ceph-osd restartd via systemd in case of disk error
On 19/09/17 10:40, Wido den Hollander wrote: > >> Op 19 september 2017 om 10:24 schreef Adrian Saul >> : >> >> >>> I understand what you mean and it's indeed dangerous, but see: >>> https://github.com/ceph/ceph/blob/master/systemd/ceph-osd%40.service >>> >>> Looking at the systemd docs it's difficult though: >>> https://www.freedesktop.org/software/systemd/man/systemd.service.ht >>> ml >>> >>> If the OSD crashes due to another bug you do want it to restart. >>> >>> But for systemd it's not possible to see if the crash was due to a disk I/O- >>> error or a bug in the OSD itself or maybe the OOM-killer or something. >> >> Perhaps using something like RestartPreventExitStatus and defining a >> specific exit code for the OSD to exit on when it is exiting due to an IO >> error. >> > > That's a very, very good idea! I didn't know that one existed. > > That would prevent restarts in case of I/O error indeed. That would depend on the OSD gracefully handling the I/O failure - IME they quite often seem to end up abort()ing... Regards, Matthew -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-osd restartd via systemd in case of disk error
I like this, there is some similar ideas we probably can borrow from Cassandra on disk failure # policy for data disk failures: # die: shut down gossip and Thrift and kill the JVM for any fs errors or # single-sstable errors, so the node can be replaced. # stop_paranoid: shut down gossip and Thrift even for single-sstable errors. # stop: shut down gossip and Thrift, leaving the node effectively dead, but # can still be inspected via JMX. # best_effort: stop using the failed disk and respond to requests based on # remaining available sstables. This means you WILL see obsolete # data at CL.ONE! # ignore: ignore fatal errors and let requests fail, as in pre-1.2 Cassandra disk_failure_policy: stop_paranoid Regards Stanley On 19/09/17 9:16 PM, Manuel Lausch wrote: Am Tue, 19 Sep 2017 08:24:48 + schrieb Adrian Saul : I understand what you mean and it's indeed dangerous, but see: https://github.com/ceph/ceph/blob/master/systemd/ceph-osd%40.service Looking at the systemd docs it's difficult though: https://www.freedesktop.org/software/systemd/man/systemd.service.ht ml If the OSD crashes due to another bug you do want it to restart. But for systemd it's not possible to see if the crash was due to a disk I/O- error or a bug in the OSD itself or maybe the OOM-killer or something. Perhaps using something like RestartPreventExitStatus and defining a specific exit code for the OSD to exit on when it is exiting due to an IO error. A other idea: The OSD daemon keeps running in a defined error state and only stops the listeners with other OSDs and the clients. -- *Stanley Zhang | * Senior Operations Engineer *Telephone:* +64 9 302 0515 *Fax:* +64 9 302 0518 *Mobile:* +64 22 318 3664 *Freephone:* 0800 SMX SMX (769 769) *SMX Limited:* Level 15, 19 Victoria Street West, Auckland, New Zealand *Web:* http://smxemail.com SMX | Cloud Email Hosting & Security _ This email has been filtered by SMX. For more info visit http://smxemail.com _ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-osd restartd via systemd in case of disk error
> Op 19 september 2017 om 10:24 schreef Adrian Saul > : > > > > I understand what you mean and it's indeed dangerous, but see: > > https://github.com/ceph/ceph/blob/master/systemd/ceph-osd%40.service > > > > Looking at the systemd docs it's difficult though: > > https://www.freedesktop.org/software/systemd/man/systemd.service.ht > > ml > > > > If the OSD crashes due to another bug you do want it to restart. > > > > But for systemd it's not possible to see if the crash was due to a disk I/O- > > error or a bug in the OSD itself or maybe the OOM-killer or something. > > Perhaps using something like RestartPreventExitStatus and defining a specific > exit code for the OSD to exit on when it is exiting due to an IO error. > That's a very, very good idea! I didn't know that one existed. That would prevent restarts in case of I/O error indeed. Wido > Confidentiality: This email and any attachments are confidential and may be > subject to copyright, legal or some other professional privilege. They are > intended solely for the attention and use of the named addressee(s). They may > only be copied, distributed or disclosed with the consent of the copyright > owner. If you have received this email by mistake or by breach of the > confidentiality clause, please notify the sender immediately by return email > and delete or destroy all copies of the email. Any confidentiality, privilege > or copyright is not waived or lost because this email has been sent to you by > mistake. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-osd restartd via systemd in case of disk error
Am Tue, 19 Sep 2017 08:24:48 + schrieb Adrian Saul : > > I understand what you mean and it's indeed dangerous, but see: > > https://github.com/ceph/ceph/blob/master/systemd/ceph-osd%40.service > > > > Looking at the systemd docs it's difficult though: > > https://www.freedesktop.org/software/systemd/man/systemd.service.ht > > ml > > > > If the OSD crashes due to another bug you do want it to restart. > > > > But for systemd it's not possible to see if the crash was due to a > > disk I/O- error or a bug in the OSD itself or maybe the OOM-killer > > or something. > > Perhaps using something like RestartPreventExitStatus and defining a > specific exit code for the OSD to exit on when it is exiting due to > an IO error. A other idea: The OSD daemon keeps running in a defined error state and only stops the listeners with other OSDs and the clients. -- Manuel Lausch Systemadministrator Cloud Services 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de Amtsgericht Montabaur, HRB 5452 Geschäftsführer: Thomas Ludwig, Jan Oetjen Member of United Internet Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-osd restartd via systemd in case of disk error
> I understand what you mean and it's indeed dangerous, but see: > https://github.com/ceph/ceph/blob/master/systemd/ceph-osd%40.service > > Looking at the systemd docs it's difficult though: > https://www.freedesktop.org/software/systemd/man/systemd.service.ht > ml > > If the OSD crashes due to another bug you do want it to restart. > > But for systemd it's not possible to see if the crash was due to a disk I/O- > error or a bug in the OSD itself or maybe the OOM-killer or something. Perhaps using something like RestartPreventExitStatus and defining a specific exit code for the OSD to exit on when it is exiting due to an IO error. Confidentiality: This email and any attachments are confidential and may be subject to copyright, legal or some other professional privilege. They are intended solely for the attention and use of the named addressee(s). They may only be copied, distributed or disclosed with the consent of the copyright owner. If you have received this email by mistake or by breach of the confidentiality clause, please notify the sender immediately by return email and delete or destroy all copies of the email. Any confidentiality, privilege or copyright is not waived or lost because this email has been sent to you by mistake. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-osd restartd via systemd in case of disk error
> Op 19 september 2017 om 10:02 schreef Manuel Lausch : > > > Hi, > > I see a issue with systemd's restart behaviour and disk IO-errors > If a disk fails with IO-errors ceph-osd stops running. Systemd detects > this and starts the daemon again. In our cluster I did see some loops > with osd crashes caused by disk failure and restarts triggerd by > systemd. Every time with peering impact and timeouts to our application > until systemd gave up. > > Obviously ceph needs the restart feature (at least with dmcrypt) to > avoid raceconditions In the startup process. But in the > case of disk related failures this is contraproductive. > > What do you think about this? Is this a bug which should be fixed? > I understand what you mean and it's indeed dangerous, but see: https://github.com/ceph/ceph/blob/master/systemd/ceph-osd%40.service Looking at the systemd docs it's difficult though: https://www.freedesktop.org/software/systemd/man/systemd.service.html If the OSD crashes due to another bug you do want it to restart. But for systemd it's not possible to see if the crash was due to a disk I/O-error or a bug in the OSD itself or maybe the OOM-killer or something. Wido > We use ceph jewel (10.2.9) > > > Regards > Manuel > > > -- > Manuel Lausch > > Systemadministrator > Cloud Services > > 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | > 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 > E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de > > Amtsgericht Montabaur, HRB 5452 > > Geschäftsführer: Thomas Ludwig, Jan Oetjen > > > Member of United Internet > > Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte > Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat > sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie > bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem > bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, > weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu > verwenden. > > This e-mail may contain confidential and/or privileged information. If > you are not the intended recipient of this e-mail, you are hereby > notified that saving, distribution or use of the content of this e-mail > in any way is prohibited. If you have received this e-mail in error, > please notify the sender and delete the e-mail. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph-osd restartd via systemd in case of disk error
Hi, I see a issue with systemd's restart behaviour and disk IO-errors If a disk fails with IO-errors ceph-osd stops running. Systemd detects this and starts the daemon again. In our cluster I did see some loops with osd crashes caused by disk failure and restarts triggerd by systemd. Every time with peering impact and timeouts to our application until systemd gave up. Obviously ceph needs the restart feature (at least with dmcrypt) to avoid raceconditions In the startup process. But in the case of disk related failures this is contraproductive. What do you think about this? Is this a bug which should be fixed? We use ceph jewel (10.2.9) Regards Manuel -- Manuel Lausch Systemadministrator Cloud Services 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de Amtsgericht Montabaur, HRB 5452 Geschäftsführer: Thomas Ludwig, Jan Oetjen Member of United Internet Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
