Re: [HACKERS] log chunking broken with large queries under load

2012-04-04 Thread Andrew Dunstan
On 04/04/2012 03:09 PM, Tom Lane wrote: I wrote: The idea I had in mind was to compensate for adding list-removal logic by getting rid of the concept of an unused entry. If the removal is conditional then you can't do that and you end up with the complications of both methods. Anyway I've no

Re: [HACKERS] log chunking broken with large queries under load

2012-04-04 Thread Tom Lane
I wrote: > The idea I had in mind was to compensate for adding list-removal logic > by getting rid of the concept of an unused entry. If the removal is > conditional then you can't do that and you end up with the complications > of both methods. Anyway I've not tried to code it yet. I concluded

Re: [HACKERS] log chunking broken with large queries under load

2012-04-04 Thread Tom Lane
Andrew Dunstan writes: > On 04/04/2012 12:13 PM, Tom Lane wrote: >> Does anyone feel that it's a bad idea that list entries are never >> reclaimed? In the worst case a transient load peak could result in >> a long list that permanently adds search overhead. Not sure if it's >> worth the extra co

Re: [HACKERS] log chunking broken with large queries under load

2012-04-04 Thread Andrew Dunstan
On 04/04/2012 12:13 PM, Tom Lane wrote: Andrew Dunstan writes: On 04/02/2012 01:03 PM, Tom Lane wrote: When I said "list", I meant a "List *". No fixed size. Ok, like this? I think this could use a bit of editorialization (I don't think the "stripe" terminology is still applicable, in par

Re: [HACKERS] log chunking broken with large queries under load

2012-04-04 Thread Tom Lane
Andrew Dunstan writes: > On 04/02/2012 01:03 PM, Tom Lane wrote: >> When I said "list", I meant a "List *". No fixed size. > Ok, like this? I think this could use a bit of editorialization (I don't think the "stripe" terminology is still applicable, in particular), but the general idea seems OK

Re: [HACKERS] log chunking broken with large queries under load

2012-04-03 Thread Andrew Dunstan
On 04/02/2012 01:03 PM, Tom Lane wrote: Andrew Dunstan writes: On 04/02/2012 12:44 PM, Tom Lane wrote: You could do something like having a list of pending chunks for each value of (pid mod 256). The length of each such list ought to be plenty short under ordinary circumstances. Yeah, ok,

Re: [HACKERS] log chunking broken with large queries under load

2012-04-02 Thread Tom Lane
Andrew Dunstan writes: > On 04/02/2012 12:44 PM, Tom Lane wrote: >> You could do something like having a list of pending chunks for each >> value of (pid mod 256). The length of each such list ought to be plenty >> short under ordinary circumstances. > Yeah, ok, that should work. How big would w

Re: [HACKERS] log chunking broken with large queries under load

2012-04-02 Thread Andrew Dunstan
On 04/02/2012 12:44 PM, Tom Lane wrote: Andrew Dunstan writes: On 04/02/2012 12:00 PM, Tom Lane wrote: This seems like it isn't actually fixing the problem, only pushing out the onset of trouble a bit. Should we not replace the fixed-size array with a dynamic data structure? But maybe your

Re: [HACKERS] log chunking broken with large queries under load

2012-04-02 Thread Tom Lane
Andrew Dunstan writes: > On 04/02/2012 12:00 PM, Tom Lane wrote: >> This seems like it isn't actually fixing the problem, only pushing out >> the onset of trouble a bit. Should we not replace the fixed-size array >> with a dynamic data structure? > But maybe your're right. If we do that and stic

Re: [HACKERS] log chunking broken with large queries under load

2012-04-02 Thread Andrew Dunstan
On 04/02/2012 12:00 PM, Tom Lane wrote: Andrew Dunstan writes: On 04/01/2012 06:34 PM, Andrew Dunstan wrote: Some of my PostgreSQL Experts colleagues have been complaining to me that servers under load with very large queries cause CSV log files that are corrupted, We could just increase CH

Re: [HACKERS] log chunking broken with large queries under load

2012-04-02 Thread Tom Lane
Andrew Dunstan writes: > On 04/01/2012 06:34 PM, Andrew Dunstan wrote: >> Some of my PostgreSQL Experts colleagues have been complaining to me >> that servers under load with very large queries cause CSV log files >> that are corrupted, > We could just increase CHUNK_SLOTS in syslogger.c, but I

Re: [HACKERS] log chunking broken with large queries under load

2012-04-02 Thread Andrew Dunstan
On 04/01/2012 06:34 PM, Andrew Dunstan wrote: Some of my PostgreSQL Experts colleagues have been complaining to me that servers under load with very large queries cause CSV log files that are corrupted, because lines are apparently multiplexed. The log chunking protocol between the errlog rou

[HACKERS] log chunking broken with large queries under load

2012-04-01 Thread Andrew Dunstan
Some of my PostgreSQL Experts colleagues have been complaining to me that servers under load with very large queries cause CSV log files that are corrupted, because lines are apparently multiplexed. The log chunking protocol between the errlog routines and the syslogger is supposed to prevent t