Hi,

At 17:54 24/04/2008, Christopher Browne wrote:
I'm not sure we want to stop queueing events altogether for any
extended period of time.  *That* seems like a risky thing to do.

I'm not quite sure why? I'd rather have them stay untouched in the DB rather than have slon grow (potentially a lot) for no good reason.

- If the problem is that there is a backlog of SYNCs that *will* need
  to be processed, then metering them in, via "LIMIT N" + some delays,
  should prevent the slon from blowing up.  If it *does* blow up, it'll
  restart, *hopefully* after getting some work done.

It does indeed, but in the meantime you have requested lots of events that ended up not being used (and which you will fetch again on the next run), you have used memory that could be more useful as OS cache, and in many cases you actually end up stopping slon quite abruptly while it's fetching data, with postgres continuing to work on that fetch while you start a new one. And really the "oh anyway it will crash and restart" approach gives be goose bumps for something related to DB replication!

The alternative solution is to do "strict metering" where we don't
allow the queue to grow past some pre-defined size.  But I'm not sure
what that size should be.

n x sync_group_maxsize? With n somewhere between 2 and 10, I'd say.

Ah, you could be right there.  Yes, it may be that the "time for first
fetch" is nearly constant, and so should be taken out of the estimate.

It's at least somewhat constant for periods of time, when the index isn't selective enough and the initial fetch needs a lot of work. Over longer periods it does vary quite significantly.

Mind you, we may be "gilding buggy whips" here; trying to improve an
estimate that is fundamentally flawed.  There is the fundamental flaw
that there is no real reason to expect two SYNCs to be equally
expensive, if there is variation in system load.

Over consecutive runs I would expect them to be quite consistent, there would just be an issue at the point where the load changes (start or end of a batch job, etc.). Obviously for a system with lots of short spikes and low values of desired_sync_time it would not make much sense, but then I'm not sure the desired_sync_time would make much sense either?

Jacques.

_______________________________________________
Slony1-general mailing list
[email protected]
http://lists.slony.info/mailman/listinfo/slony1-general

Reply via email to