If we are working on cron can I inject a feature request? Support for multiple crontab files in applications/<myapp>/cron
crontab, or plugin_hi.crontab can be read. Alpha order ? -- Thadeus On Tue, Apr 13, 2010 at 10:47 AM, AchipA <attila.cs...@gmail.com> wrote: > Technically, we don't need to nor are we required to skip *exactly* > one minute. The real question is not 'has 60 seconds passed', but > 'have we checked cron in this minute'. I would thus check not for a 60 > sec offset, but rather if the minute of cron start matches. > > Also in line 51: > > s.enter(60 - now % 60, 1, self.launch, ()) > > it would probably be better to use int(now) to avoid the sleep/time > functions rounding/timing errors potentially result in a 59.99... > > On Apr 13, 3:18 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: >> I do not like very much the idea of setting ctime/mtime but I do not >> like the current mechanism either. It is not just the pickle overhead. >> There seem to be precision issues that make it difficult to skip >> exactly one minute. For example right now newcron contains: >> >> if startup or self.now - start > 59.99 >> >> 59.99 should be 60 but that causes cron to skip some calls. 59.99 is >> OK most of the cases but it may still cause some false positives. >> >> I do not know have a simple solution to this problem. It seems to me >> we need to store more info, not less, even start/stop times may not be >> sufficient information. We may also need to id each schedules call to >> cron and detect not only whether a cron task is running but which of >> past scheduled tasks are running. >> >> Bottom line: the part of cron/newcron that runs every 1 minute need to >> be redesigned completely in my view. >> >> Massimo >> >> On Apr 13, 6:23 am, AchipA <attila.cs...@gmail.com> wrote: >> >> > Hey, I don't have a problem with that, just saying the lock-and-read >> > mechanism can increase the overhead/latency significantly in certain >> > setups. If a two-file approach is a problem, would you consider >> > avoiding the cPickle read if I can find a way to do it via setting >> > ctime/mtime (we set a mtime older then ctime, and when we finish, we >> > set now() as mtime) ? >> >> > Alternatively, if it turns out that can't be done on all platforms and >> > we only need to check when it started and whether it is running, we >> > could simplify running checks by file size. When the cron starts, we >> > create the file with zero size, and when it finishes we just write a >> > space to it. That way we can check whether it's running or not just by >> > looking at the file size (a lot cheaper that reading/pickling). >> >> > The bottom line is that I would like to avoid having to do locking for >> > the read-checks (we write only once a minute, but read quite a bit >> > more as we scale upwards - like on a shared volume or on a busy site >> > with softcron). >> >> > On Apr 12, 11:27 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: >> >> > > On Apr 12, 1:19 pm, AchipA <attila.cs...@gmail.com> wrote: >> >> > > > Why do we need the time range ? If the tasks are overlapping it's >> > > > their responsibility to handle that (I know this is arguable, but >> > > > that's how 'standard' cron works). >> >> > > This is also as it works in newcron. The problem is that if for any >> > > reason, the main process that loops and spans the tasks gets stuck, it >> > > may give rise to a proliferation of processes that may crash the os. >> > > The current mechanism is similar to the one you originally implemented >> > > but you used a n additional file to determine if the cron was >> > > completed. I use the completion date. >> >> > > > Also, we can easily store two >> > > > timestamps (slightly hackish, but mtime and ctime can be set >> > > > separately), would have to check whether that is supported on all >> > > > platforms. Of course there are many other ways of reading data without >> > > > opening files, I'm just pondering about alternatives as the current >> > > > locking mechanism causes some problems on my shared-volume based multi- >> > > > server setup (that's why I used 'move' originally as it's atomic and >> > > > works well with netwok shares). >> >> > > > On Apr 12, 5:12 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: >> >> > > > > Because they os timestamp only can only tell you when a task has >> > > > > started (or stopped, depending on when it was created) it does not >> > > > > contain enough information to give you a time range (time and stop). >> > > > > Cron needs to know when the previous crondance started and whether is >> > > > > was completed or not. The original implementation was doing the check >> > > > > using locks and that resulted in a large number of try... except... >> > > > > The current implementation removes most of the try.. except... >> > > > > (people >> > > > > complained about that) and just stores start_time, stop_time >> > > > > explicitly in a picke. >> >> > > > > On Apr 12, 8:00 am, AchipA <attila.cs...@gmail.com> wrote: >> >> > > > > > To correct myself, it seems the cron in web2py no longer uses the >> > > > > > filesystem timestamps, but cPickles timestamps from/to the lock >> > > > > > file. >> > > > > > I'm not sure why Massimo changed it, but this *is* a bigger >> > > > > > overhead >> > > > > > than it was previously (as it needs to do file locking and >> > > > > > cPickle.load() on every single request - as opposed to a simple >> > > > > > cached >> > > > > > non-locking filesystem call). >> >> > > > > > On Apr 1, 8:20 pm, AchipA <attila.cs...@gmail.com> wrote: >> >> > > > > > > Exactly, hardcron checks once a minute, softcron checks on each >> > > > > > > page >> > > > > > > load. The 'check' is calling a function or two and comparing a >> > > > > > > file's >> > > > > > > timestamp, so not *that* much more expensive. >> >> > > > > > > On Apr 1, 7:51 pm, Jonathan Lundell <jlund...@pobox.com> wrote: >> >> > > > > > > > On Apr 1, 2010, at 10:37 AM, AchipA wrote: >> >> > > > > > > > > There is some overhead, but efficiency is a disputable term >> > > > > > > > > - there is >> > > > > > > > > certainly more overhead than hardcron, but IMO not in a way >> > > > > > > > > that would >> > > > > > > > > affect overall performance unless you're running it on a >> > > > > > > > > site that has >> > > > > > > > > hundreds of thousands of hits per day... >> >> > > > > > > > Perhaps we could change (or eliminate) the wording. How about >> > > > > > > > simply 'Using softcron'? >> >> > > > > > > > I'm curious: what is the extra overhead of soft vs hardcron? >> > > > > > > > Just that it does a test on each page access? I'm guessing >> > > > > > > > that's pretty cheap. >> >> > > > > > > > > On Apr 1, 5:40 pm, Jonathan Lundell <jlund...@pobox.com> >> > > > > > > > > wrote: >> > > > > > > > >> Section 4.17 (cron) mentions hard vs softcrondefaults, but >> > > > > > > > >> doesn't say how to override them. >> >> > > > > > > > >> Section 4.1 (cli) doesn't list --softcron >> >> > > > > > > > >> The startup message for softcronsays: 'Using softcron (but >> > > > > > > > >> this is not very efficient)' >> >> > > > > > > > >> In what sense "not efficient"? I understand that the timing >> > > > > > > > >> is less consistent, but is there really more overhead? >> > > > > > > > >> softcron seems like a pretty reasonable choice if all >> > > > > > > > >> you're doing it deleting expired sessions. >> >> > > > -- > To unsubscribe, reply using "remove me" as the subject. >
