Re: [CentOS] CentOS 8.1 cron does not send mail
On 6 Apr 2020, at 12:21, Stephen John Smoogen wrote: On Mon, 6 Apr 2020 at 04:16, Tobias Kirchhofer wrote: On 5 Apr 2020, at 21:20, Tobias Kirchhofer wrote: we experience difficulties with crond behaviour sending mail since CentOS 8.1. The cron job is the same like we used in CentOS 7. Meanwhile we found the reason for the bug - actually we do not know if it is related to a specific version of CentOS or a specific kind of command as cron job. Let me explain what we have: - sssd for ssh login of ldap user - crond for cron jobs :) If we stop sssd and restart crond cron starts to send mails again! We started with sssd on newly provisioned machines with CentOS 8. We do not know if this is the same on CentOS 7. We send mails only to root. So no remote user is involved in cron. From our perspective it is a bug. How could we dive deeper to find the specific reason? To sum it up: - Install CentOS 8 - Enabled and started crond - crond sends emails properly - Enable and start sssd - crond stops sending emails and starts journal logging - Restart crond (or reboot) - crond sends emails and stops journal logging It is a matter of order. At boot time crond starts after sssd. This situation is bearable if you know it but has cost us some hours. Thanks for reading and sorry for this public clarification process ;) Tobias So it sounds like that crond needs to have sssd as a pre-dependency so it doesn't start until sssd is running? No - if crond is already running and sssd is initially set and starting (after crond) crond does not send mail. For whatever reason. At boot time things are okay, crond starts after sssd. So if sssd is already there, crond is fine. If sssd starts after crond, crond is not fine. It is a kind of race condition which takes place only in the beginning when sssd is set. Usually crond is in this use case already running, because it is part of (our) base install. Hope this is clear :) -- collect@shift.agency ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS 8.1 cron does not send mail
On 5 Apr 2020, at 21:20, Tobias Kirchhofer wrote:
we experience difficulties with crond behaviour sending mail since
CentOS 8.1. The cron job is the same like we used in CentOS 7.
Meanwhile we found the reason for the bug - actually we do not know if
it is related to a specific version of CentOS or a specific kind of
command as cron job.
Let me explain what we have:
- sssd for ssh login of ldap user
- crond for cron jobs :)
If we stop sssd and restart crond cron starts to send mails again!
We started with sssd on newly provisioned machines with CentOS 8. We
do not know if this is the same on CentOS 7.
We send mails only to root. So no remote user is involved in cron.
From our perspective it is a bug. How could we dive deeper to find the
specific reason?
To sum it up:
- Install CentOS 8
- Enabled and started crond
- crond sends emails properly
- Enable and start sssd
- crond stops sending emails and starts journal logging
- Restart crond (or reboot)
- crond sends emails and stops journal logging
It is a matter of order. At boot time crond starts after sssd.
This situation is bearable if you know it but has cost us some hours.
Thanks for reading and sorry for this public clarification process ;)
Tobias
What do you think about this?
Tobias
On 21 Mar 2020, at 11:56, Tobias Kirchhofer wrote:
On 27 Feb 2020, at 14:42, Jonathan Billings wrote:
On Feb 27, 2020, at 08:01, Tobias Kirchhofer
wrote:
Hi,
we experience difficulties with crond behaviour sending mail since
CentOS 8.1. The cron job is the same like we used in CentOS 7.
crontab -l
/usr/bin/python3 -c 'import random; import time;
time.sleep(random.random() * 3600)' && /usr/local/bin/backup.sh
Is this literally what your crontab looks like? Because that’s
not valid crontab syntax.
This is what it is literally:
0 5 * * * /usr/bin/python3 -c 'import random; import time;
time.sleep(random.random() * 3600)' && /usr/local/bin/backup.sh
backup.sh writes the backup to the remote backup server. There are
around 30 machines with the same cron job. The python part spreads
the jobs over an hour so that the backup server is not struck at
once from about 30 machines.
The change in behaviour is that crond sends no mail anymore. It is
because of the logical and (&&). Without this crond sends mails.
What has changed in CentOS 8 and does anyone has an idea how we
could fix it?
There aren’t any significant changes in ‘cronie’ in 8.1,
looking at the spec file.
Assuming the crontab you wrote above included the time spec too,
I’d check to make sure the first command isn’t exiting with a
non-zero exit code.
The command chain is running propery, STDOUT output is visible in
system log (excerpt from logwatch email):
```
- Cron Begin
**Unmatched Entries**
CMDOUT (### Starting backup. Host: host.example.com Backupserver:
10.9.1.5 Path: /borgbackup/vm/host-example-com Date: Fri Mar 20
05:13:46 CET 2020 ###)
CMDOUT (Creating archive at
"borg@IP:/borgbackup/vm/host-example-com::{now:%Y-%m-%d_%H:%M}")
…
```
Its just not sending an email.
What is additionally irritating is that it is only on a few machines.
All machines are identically provisioned with ansible.
The situation is not mission critical but it should work. Strange.
Tobias
--
collect@shift.agency
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS 8.1 cron does not send mail
we experience difficulties with crond behaviour sending mail since
CentOS 8.1. The cron job is the same like we used in CentOS 7.
Meanwhile we found the reason for the bug - actually we do not know if
it is related to a specific version of CentOS or a specific kind of
command as cron job.
Let me explain what we have:
- sssd for ssh login of ldap user
- crond for cron jobs :)
If we stop sssd and restart crond cron starts to send mails again!
We started with sssd on newly provisioned machines with CentOS 8. We do
not know if this is the same on CentOS 7.
We send mails only to root. So no remote user is involved in cron.
From our perspective it is a bug. How could we dive deeper to find the
specific reason?
What do you think about this?
Tobias
On 21 Mar 2020, at 11:56, Tobias Kirchhofer wrote:
On 27 Feb 2020, at 14:42, Jonathan Billings wrote:
On Feb 27, 2020, at 08:01, Tobias Kirchhofer
wrote:
Hi,
we experience difficulties with crond behaviour sending mail since
CentOS 8.1. The cron job is the same like we used in CentOS 7.
crontab -l
/usr/bin/python3 -c 'import random; import time;
time.sleep(random.random() * 3600)' && /usr/local/bin/backup.sh
Is this literally what your crontab looks like? Because that’s
not valid crontab syntax.
This is what it is literally:
0 5 * * * /usr/bin/python3 -c 'import random; import time;
time.sleep(random.random() * 3600)' && /usr/local/bin/backup.sh
backup.sh writes the backup to the remote backup server. There are
around 30 machines with the same cron job. The python part spreads
the jobs over an hour so that the backup server is not struck at
once from about 30 machines.
The change in behaviour is that crond sends no mail anymore. It is
because of the logical and (&&). Without this crond sends mails.
What has changed in CentOS 8 and does anyone has an idea how we
could fix it?
There aren’t any significant changes in ‘cronie’ in 8.1,
looking at the spec file.
Assuming the crontab you wrote above included the time spec too,
I’d check to make sure the first command isn’t exiting with a
non-zero exit code.
The command chain is running propery, STDOUT output is visible in
system log (excerpt from logwatch email):
```
- Cron Begin
**Unmatched Entries**
CMDOUT (### Starting backup. Host: host.example.com Backupserver:
10.9.1.5 Path: /borgbackup/vm/host-example-com Date: Fri Mar 20
05:13:46 CET 2020 ###)
CMDOUT (Creating archive at
"borg@IP:/borgbackup/vm/host-example-com::{now:%Y-%m-%d_%H:%M}")
…
```
Its just not sending an email.
What is additionally irritating is that it is only on a few machines.
All machines are identically provisioned with ansible.
The situation is not mission critical but it should work. Strange.
Tobias
--
collect@shift.agency
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] CentOS 8.1 cron does not send mail
On 27 Feb 2020, at 14:42, Jonathan Billings wrote:
On Feb 27, 2020, at 08:01, Tobias Kirchhofer
wrote:
Hi,
we experience difficulties with crond behaviour sending mail since
CentOS 8.1. The cron job is the same like we used in CentOS 7.
crontab -l
/usr/bin/python3 -c 'import random; import time;
time.sleep(random.random() * 3600)' && /usr/local/bin/backup.sh
Is this literally what your crontab looks like? Because that’s not
valid crontab syntax.
This is what it is literally:
0 5 * * * /usr/bin/python3 -c 'import random; import time;
time.sleep(random.random() * 3600)' && /usr/local/bin/backup.sh
backup.sh writes the backup to the remote backup server. There are
around 30 machines with the same cron job. The python part spreads
the jobs over an hour so that the backup server is not struck at once
from about 30 machines.
The change in behaviour is that crond sends no mail anymore. It is
because of the logical and (&&). Without this crond sends mails.
What has changed in CentOS 8 and does anyone has an idea how we could
fix it?
There aren’t any significant changes in ‘cronie’ in 8.1, looking
at the spec file.
Assuming the crontab you wrote above included the time spec too, I’d
check to make sure the first command isn’t exiting with a non-zero
exit code.
The command chain is running propery, STDOUT output is visible in system
log (excerpt from logwatch email):
```
- Cron Begin
**Unmatched Entries**
CMDOUT (### Starting backup. Host: host.example.com Backupserver:
10.9.1.5 Path: /borgbackup/vm/host-example-com Date: Fri Mar 20 05:13:46
CET 2020 ###)
CMDOUT (Creating archive at
"borg@IP:/borgbackup/vm/host-example-com::{now:%Y-%m-%d_%H:%M}")
…
```
Its just not sending an email.
What is additionally irritating is that it is only on a few machines.
All machines are identically provisioned with ansible.
The situation is not mission critical but it should work. Strange.
Tobias
--
Jonathan Billings
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos
--
collect@shift.agency
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Borgbackup question
On 2 Mar 2020, at 13:35, Alessandro Baggi wrote: Il 02/03/20 13:18, Tobias Kirchhofer ha scritto: On 2 Mar 2020, at 12:58, Alessandro Baggi wrote: Il 01/03/20 20:18, Tobias Kirchhofer ha scritto: On 1 Mar 2020, at 20:00, Gordon Messmer wrote: On 3/1/20 12:40 AM, Alessandro Baggi wrote: borgbackup is a very interesting backup tool with a lot of features. It is ready for "production" or I should expect some bad surprise? I don't know the answer to that, but to me that implies two questions: 1) Are there failure conditions that it doesn't handle, especially with an interrupted backup, and 2) Does it perform poorly under any specific circumstances. If anyone has experience with those questions, or is familiar enough with the implementation to explain why those should not be an issue, I'd be interested in their input as well. I use borgbackup for several laptops backing up to a local file server with sshfs, and that's been good so far. We have around 50 linux clients with borg backups to two different backup server, provisioned with Ansible. A new host is in the backup in around 30 seconds :) One backup server is internal for DMZ and LAN and one is for external hosts. The internal backup server syncs its backup to the external server. Storage is made with ZFS summed up to 16 TB each server. This runs nicely for around two years without interruption. We learned a bit her e and there about some side effects with borg cache in the beginning and invested some time in hardening and Ansible role. Before we choosed borg restic was on the list. Looks good too. Do not now anymore why we decided for borg. Maybe the name :) We startet here https://borgbackup.readthedocs.io/en/stable/deployment/central-backup-server.html Tobias Hi Tobias, How do you secure the process? Plain ssh: authorized_keys on the backup server: ``` … command="borg serve --restrict-to-path /borgbackup/vm/host-name-of-backup-client --append-only" ssh-ed25519 C3NzaC1… root@host-name-of-backup-client … ``` Ok thank you but how do you run prune command if append-only is enabled? Prune is startet on the client after each backup. Before we did it on the backup server at once. But borg recreated the whole index per repo each time. The mailing list was helpful with this. borg prune must run on the machine where the backup is created. append-only is not involved in borg prune. -- collect@shift.agency ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Borgbackup question
On 2 Mar 2020, at 12:58, Alessandro Baggi wrote: Il 01/03/20 20:18, Tobias Kirchhofer ha scritto: On 1 Mar 2020, at 20:00, Gordon Messmer wrote: On 3/1/20 12:40 AM, Alessandro Baggi wrote: borgbackup is a very interesting backup tool with a lot of features. It is ready for "production" or I should expect some bad surprise? I don't know the answer to that, but to me that implies two questions: 1) Are there failure conditions that it doesn't handle, especially with an interrupted backup, and 2) Does it perform poorly under any specific circumstances. If anyone has experience with those questions, or is familiar enough with the implementation to explain why those should not be an issue, I'd be interested in their input as well. I use borgbackup for several laptops backing up to a local file server with sshfs, and that's been good so far. We have around 50 linux clients with borg backups to two different backup server, provisioned with Ansible. A new host is in the backup in around 30 seconds :) One backup server is internal for DMZ and LAN and one is for external hosts. The internal backup server syncs its backup to the external server. Storage is made with ZFS summed up to 16 TB each server. This runs nicely for around two years without interruption. We learned a bit her e and there about some side effects with borg cache in the beginning and invested some time in hardening and Ansible role. Before we choosed borg restic was on the list. Looks good too. Do not now anymore why we decided for borg. Maybe the name :) We startet here https://borgbackup.readthedocs.io/en/stable/deployment/central-backup-server.html Tobias Hi Tobias, How do you secure the process? On more thing - the borg mailing list is responsive. I had a question. Answer arrived directly https://www.borgbackup.org/support/free.html -- collect@shift.agency ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Borgbackup question
On 2 Mar 2020, at 12:58, Alessandro Baggi wrote: Il 01/03/20 20:18, Tobias Kirchhofer ha scritto: On 1 Mar 2020, at 20:00, Gordon Messmer wrote: On 3/1/20 12:40 AM, Alessandro Baggi wrote: borgbackup is a very interesting backup tool with a lot of features. It is ready for "production" or I should expect some bad surprise? I don't know the answer to that, but to me that implies two questions: 1) Are there failure conditions that it doesn't handle, especially with an interrupted backup, and 2) Does it perform poorly under any specific circumstances. If anyone has experience with those questions, or is familiar enough with the implementation to explain why those should not be an issue, I'd be interested in their input as well. I use borgbackup for several laptops backing up to a local file server with sshfs, and that's been good so far. We have around 50 linux clients with borg backups to two different backup server, provisioned with Ansible. A new host is in the backup in around 30 seconds :) One backup server is internal for DMZ and LAN and one is for external hosts. The internal backup server syncs its backup to the external server. Storage is made with ZFS summed up to 16 TB each server. This runs nicely for around two years without interruption. We learned a bit her e and there about some side effects with borg cache in the beginning and invested some time in hardening and Ansible role. Before we choosed borg restic was on the list. Looks good too. Do not now anymore why we decided for borg. Maybe the name :) We startet here https://borgbackup.readthedocs.io/en/stable/deployment/central-backup-server.html Tobias Hi Tobias, How do you secure the process? Plain ssh: authorized_keys on the backup server: ``` … command="borg serve --restrict-to-path /borgbackup/vm/host-name-of-backup-client --append-only" ssh-ed25519 C3NzaC1… root@host-name-of-backup-client … ``` -- collect@shift.agency ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Borgbackup question
On 1 Mar 2020, at 20:00, Gordon Messmer wrote: On 3/1/20 12:40 AM, Alessandro Baggi wrote: borgbackup is a very interesting backup tool with a lot of features. It is ready for "production" or I should expect some bad surprise? I don't know the answer to that, but to me that implies two questions: 1) Are there failure conditions that it doesn't handle, especially with an interrupted backup, and 2) Does it perform poorly under any specific circumstances. If anyone has experience with those questions, or is familiar enough with the implementation to explain why those should not be an issue, I'd be interested in their input as well. I use borgbackup for several laptops backing up to a local file server with sshfs, and that's been good so far. We have around 50 linux clients with borg backups to two different backup server, provisioned with Ansible. A new host is in the backup in around 30 seconds :) One backup server is internal for DMZ and LAN and one is for external hosts. The internal backup server syncs its backup to the external server. Storage is made with ZFS summed up to 16 TB each server. This runs nicely for around two years without interruption. We learned a bit her e and there about some side effects with borg cache in the beginning and invested some time in hardening and Ansible role. Before we choosed borg restic was on the list. Looks good too. Do not now anymore why we decided for borg. Maybe the name :) We startet here https://borgbackup.readthedocs.io/en/stable/deployment/central-backup-server.html Tobias -- collect@shift.agency ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
[CentOS] CentOS 8.1 cron does not send mail
Hi, we experience difficulties with crond behaviour sending mail since CentOS 8.1. The cron job is the same like we used in CentOS 7. crontab -l /usr/bin/python3 -c 'import random; import time; time.sleep(random.random() * 3600)' && /usr/local/bin/backup.sh backup.sh writes the backup to the remote backup server. There are around 30 machines with the same cron job. The python part spreads the jobs over an hour so that the backup server is not struck at once from about 30 machines. The change in behaviour is that crond sends no mail anymore. It is because of the logical and (&&). Without this crond sends mails. What has changed in CentOS 8 and does anyone has an idea how we could fix it? Thank you! Tobias -- collect@shift.agency ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
