Re: spamd-setup fails from cron
Ok;
After running that a few days, it works fine, but... the interval between
updates
is all over the place.
I rewrote it, to only change the sleep value under 2 circumstances:
First time run, or after a failure.
Now it's updating hourly again.
I will not make the same mistake of posting it to the list, because archiving a
possibly buggy script that someone may copy someday is not a great idea.
However I think the methodology is now sound, so write your own or mail me
directly
if you want a copy of it to adopt.
On Fri, Jun 01, 2012 at 04:45:24PM +1000, David Diggles wrote:
> > #!/bin/sh
> > remaining=$1;shift
> > cmd=$@
> > lock=/var/run/$(basename $1).lock
> > [ -f $lock ] || {
> > touch $lock
> > while [ $remaining -gt 0 ]; do
> > seconds=$(($RANDOM % $remaining))
> > echo $(date) $seconds >> $lock
> > sleep $seconds
> > $cmd && return || remaining=$(($remaining - $seconds))
> > done
> > rm $lock
> > }
> >
>
> *groan*.. another mistake.. I'm such an idiot sometimes ;-)
>
> I don't recommend running this without checking it first.
>
> #!/bin/sh
> remaining=$1;shift
> cmd=$@
> lock=/var/run/$(basename $1).lock
> [ -f $lock ] || {
> touch $lock
> while [ $remaining -gt 0 ]; do
> seconds=$(($RANDOM % $remaining))
> echo $(date) $seconds >> $lock
> sleep $seconds
> $cmd && break || remaining=$(($remaining - $seconds))
> done
> rm $lock
> }
Re: spamd-setup fails from cron
> #!/bin/sh
> remaining=$1;shift
> cmd=$@
> lock=/var/run/$(basename $1).lock
> [ -f $lock ] || {
> touch $lock
> while [ $remaining -gt 0 ]; do
> seconds=$(($RANDOM % $remaining))
> echo $(date) $seconds >> $lock
> sleep $seconds
> $cmd && return || remaining=$(($remaining - $seconds))
> done
> rm $lock
> }
>
*groan*.. another mistake.. I'm such an idiot sometimes ;-)
I don't recommend running this without checking it first.
#!/bin/sh
remaining=$1;shift
cmd=$@
lock=/var/run/$(basename $1).lock
[ -f $lock ] || {
touch $lock
while [ $remaining -gt 0 ]; do
seconds=$(($RANDOM % $remaining))
echo $(date) $seconds >> $lock
sleep $seconds
$cmd && break || remaining=$(($remaining - $seconds))
done
rm $lock
}
Re: spamd-setup fails from cron
On Fri, Jun 01, 2012 at 03:47:21PM +1000, David Diggles wrote:
[ snip ]
> sleep $s
[ snip ]
Arghh.. ;-) sleep $seconds here
Anyway, you get the idea.
#!/bin/sh
remaining=$1;shift
cmd=$@
lock=/var/run/$(basename $1).lock
[ -f $lock ] || {
touch $lock
while [ $remaining -gt 0 ]; do
seconds=$(($RANDOM % $remaining))
echo $(date) $seconds >> $lock
sleep $seconds
$cmd && return || remaining=$(($remaining - $seconds))
done
rm $lock
}
Re: spamd-setup fails from cron
On Tue, May 29, 2012 at 09:51:54AM +0200, Bret Lambert wrote:
> > Please avoid 15 minutes past the hour ;-)
>
> sleep $(($RANDOM % 2048)) && /usr/libexec/spamd-setup -d
Tried something like the above, and found it still
fails at peak times, so I am trying something else:
I made a wrapper called ss (spamd sync), to keep
retrying within a diminishing timeframe.
#!/bin/sh
remaining=$1;shift
cmd=$@
lock=/var/run/$(basename $1).lock
[ -f $lock ] || {
touch $lock
while [ $remaining -gt 0 ]; do
seconds=$(($RANDOM % $remaining))
echo $(date) $seconds >> $lock
sleep $s
$cmd && return || remaining=$(($remaining - $seconds))
done
rm $lock
}
0 * * * * ss 3600 /usr/libexec/spamd-setup -d
The time overhead of running the command creates a small
possibility for overlapping of cron events, so I created
a lock file. This also seemed a good place to store the
sleep value(s).
/var/run/spamd-setup.lock
spamd-setup(8) does not say how it behaves when daemonized.
Maybe this is a better option than running from the cron?
.d.d.
Re: spamd-setup fails from cron
On May 29 09:14:29, Peter N. M. Hansteen wrote: > On Tue, May 29, 2012 at 08:24:07AM +0200, Jan Stary wrote: > > > When I run the same command from the command line, > > everything goes fine. Is the cron job run in a more > > restricted environment? > > you could be hitting the 'zero minute rush', where world+dog tries to > connect simultaneously. try shifting to a few minutes past the hour and > see if that helps. Yes it did help. Thanks! Jan
Re: spamd-setup fails from cron
On Tue, May 29, 2012 at 9:50 AM, Mitja MuE>eniD wrote: > Years ago I've been toying with the idea of having a flag for random-delay > mode in spamd-setup. So the default cronjob is still set at zero minute, but > spamd-setup then waits for random amount of minutes before hitting the > server. You lose the predictability of when your blacklists get updated but > it spreads the load on the infrastructure without involving the human factor > (the lazy admin that uncomments the cron entry without changing the time). > portsnap, a tool that FreeBSD uses for fetching the ports tree is using a simular method. Take a look here: http://svnweb.freebsd.org/base/head/usr.sbin/portsnap/portsnap/portsnap.sh?re vision=235208&view=markup -- chs,
Re: spamd-setup fails from cron
> > you could be hitting the 'zero minute rush', where world+dog tries to > > connect simultaneously. try shifting to a few minutes past the hour > and > > see if that helps. > > > > Please avoid 15 minutes past the hour ;-) Years ago I've been toying with the idea of having a flag for random-delay mode in spamd-setup. So the default cronjob is still set at zero minute, but spamd-setup then waits for random amount of minutes before hitting the server. You lose the predictability of when your blacklists get updated but it spreads the load on the infrastructure without involving the human factor (the lazy admin that uncomments the cron entry without changing the time). Mitja
Re: spamd-setup fails from cron
> Please avoid 15 minutes past the hour ;-) sleep $(($RANDOM % 2048)) && /usr/libexec/spamd-setup -d
Re: spamd-setup fails from cron
A random sleep between 0 and 3599 prior to running spamd-setup in cron would not go astray. On Tue, May 29, 2012 at 09:23:43AM +0200, Gilles Chehade wrote: > On Tue, May 29, 2012 at 09:14:29AM +0200, Peter N. M. Hansteen wrote: > > On Tue, May 29, 2012 at 08:24:07AM +0200, Jan Stary wrote: > > > > > When I run the same command from the command line, > > > everything goes fine. Is the cron job run in a more > > > restricted environment? > > > > you could be hitting the 'zero minute rush', where world+dog tries to > > connect simultaneously. try shifting to a few minutes past the hour and > > see if that helps. > > > > Please avoid 15 minutes past the hour ;-) > > -- > Gilles Chehade > > https://www.poolp.org | http://pool.ps > @poolpOrg
Re: spamd-setup fails from cron
On Tue, May 29, 2012 at 09:14:29AM +0200, Peter N. M. Hansteen wrote: > On Tue, May 29, 2012 at 08:24:07AM +0200, Jan Stary wrote: > > > When I run the same command from the command line, > > everything goes fine. Is the cron job run in a more > > restricted environment? > > you could be hitting the 'zero minute rush', where world+dog tries to > connect simultaneously. try shifting to a few minutes past the hour and > see if that helps. > Please avoid 15 minutes past the hour ;-) -- Gilles Chehade https://www.poolp.org | http://pool.ps @poolpOrg
Re: spamd-setup fails from cron
On Tue, May 29, 2012 at 08:24:07AM +0200, Jan Stary wrote: > When I run the same command from the command line, > everything goes fine. Is the cron job run in a more > restricted environment? you could be hitting the 'zero minute rush', where world+dog tries to connect simultaneously. try shifting to a few minutes past the hour and see if that helps. - Peter -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ "Remember to set the evil bit on all malicious network traffic" delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: spamd-setup fails from cron
Change it to this: * * * * /usr/libexec/spamd-setup -d It will probably fix the problem. On Tue, May 29, 2012 at 08:24:07AM +0200, Jan Stary wrote: > Pretty current 5.1-current/amd64. > This is what happens with the following line in root's crontab > > 0 * * * * /usr/libexec/spamd-setup -d > > On May 29 03:00:02, Cron Daemon wrote: > > Getting http://www.openbsd.org/spamd/traplist.gz > > spamd-setup: Could not add blacklist uatraps: Illegal seek > > Getting http://www.openbsd.org/spamd/nixspam.gz > > ftp: Writing -: Broken pipe > > blacklist nixspam 4 entries > > What is the 'illegal seek' spamd-setup reports? > What is the ftp's broken pipe? > > When I run the same command from the command line, > everything goes fine. Is the cron job run in a more > restricted environment? > > Jan
spamd-setup fails from cron
Pretty current 5.1-current/amd64. This is what happens with the following line in root's crontab 0 * * * * /usr/libexec/spamd-setup -d On May 29 03:00:02, Cron Daemon wrote: > Getting http://www.openbsd.org/spamd/traplist.gz > spamd-setup: Could not add blacklist uatraps: Illegal seek > Getting http://www.openbsd.org/spamd/nixspam.gz > ftp: Writing -: Broken pipe > blacklist nixspam 4 entries What is the 'illegal seek' spamd-setup reports? What is the ftp's broken pipe? When I run the same command from the command line, everything goes fine. Is the cron job run in a more restricted environment? Jan
