Re: [PATCHES] further WIP for COPYable logs
Here's my latest WIP patch for COPYable CSV logs. I think I addresses
all (or most :-) ) of the concerns Dave and Tom had. It's a whole lot
simpler than before (no extra pipe).
I have not yet tested this on Windows, but given that it now contains
almost no Windows-specific code I'm not so worried about it - will
test when I get a chance.
Windows tests (make installcheck-parallel with csv logs turned on)
seem to work just fine.
updated in attached to fix a couple of small logic bugs in picking the
destination file.
I think this is close to ready for documentation and application.
cheers
andrew
Index: src/backend/postmaster/syslogger.c
===
RCS file: /cvsroot/pgsql/src/backend/postmaster/syslogger.c,v
retrieving revision 1.36
diff -c -r1.36 syslogger.c
*** src/backend/postmaster/syslogger.c 4 Aug 2007 01:26:53 - 1.36
--- src/backend/postmaster/syslogger.c 8 Aug 2007 16:32:19 -
***
*** 87,93
--- 87,95
static pg_time_t next_rotation_time;
static bool pipe_eof_seen = false;
static FILE *syslogFile = NULL;
+ static FILE *csvlogFile = NULL;
static char *last_file_name = NULL;
+ static char *last_csvfile_name = NULL;
/*
* Buffers for saving partial messages from different backends. We don't expect
***
*** 132,143
#endif
static void process_pipe_input(char *logbuffer, int *bytes_in_logbuffer);
static void flush_pipe_input(char *logbuffer, int *bytes_in_logbuffer);
#ifdef WIN32
static unsigned int __stdcall pipeThread(void *arg);
#endif
! static void logfile_rotate(bool time_based_rotation);
! static char *logfile_getname(pg_time_t timestamp);
static void set_next_rotation_time(void);
static void sigHupHandler(SIGNAL_ARGS);
static void sigUsr1Handler(SIGNAL_ARGS);
--- 134,146
#endif
static void process_pipe_input(char *logbuffer, int *bytes_in_logbuffer);
static void flush_pipe_input(char *logbuffer, int *bytes_in_logbuffer);
+ static void open_csvlogfile(void);
#ifdef WIN32
static unsigned int __stdcall pipeThread(void *arg);
#endif
! static void logfile_rotate(bool time_based_rotation, int size_rotation_for);
! static char *logfile_getname(pg_time_t timestamp, char * suffix);
static void set_next_rotation_time(void);
static void sigHupHandler(SIGNAL_ARGS);
static void sigUsr1Handler(SIGNAL_ARGS);
***
*** 281,287
for (;;)
{
bool time_based_rotation = false;
!
#ifndef WIN32
int bytesRead;
int rc;
--- 284,290
for (;;)
{
bool time_based_rotation = false;
! int size_rotation_for = 0;
#ifndef WIN32
int bytesRead;
int rc;
***
*** 336,346
{
/* Do a rotation if file is too big */
if (ftell(syslogFile) = Log_RotationSize * 1024L)
rotation_requested = true;
}
if (rotation_requested)
! logfile_rotate(time_based_rotation);
#ifndef WIN32
--- 339,358
{
/* Do a rotation if file is too big */
if (ftell(syslogFile) = Log_RotationSize * 1024L)
+ {
+ rotation_requested = true;
+ size_rotation_for |= LOG_DESTINATION_STDERR;
+ }
+ if (csvlogFile != NULL ftell(csvlogFile) = Log_RotationSize * 1024L)
+ {
rotation_requested = true;
+ size_rotation_for |= LOG_DESTINATION_CSVLOG;
+ }
+
}
if (rotation_requested)
! logfile_rotate(time_based_rotation, size_rotation_for);
#ifndef WIN32
***
*** 405,411
if (pipe_eof_seen)
{
! ereport(LOG,
(errmsg(logger shutting down)));
/*
--- 417,426
if (pipe_eof_seen)
{
! /* seeing this message on the real stderr is annoying - so we
! * make it DEBUG1 to suppress in normal use.
! */
! ereport(DEBUG1,
(errmsg(logger shutting down)));
/*
***
*** 477,483
* The initial logfile is created right in the postmaster, to verify that
* the Log_directory is writable.
*/
! filename = logfile_getname(time(NULL));
syslogFile = fopen(filename, a);
--- 492,498
* The initial logfile is created right in the postmaster, to verify that
* the Log_directory is writable.
*/
! filename = logfile_getname(time(NULL), NULL);
syslogFile = fopen(filename, a);
***
*** 688,693
--- 703,709
{
char *cursor = logbuffer;
int count = *bytes_in_logbuffer;
+ int dest = LOG_DESTINATION_STDERR;
/* While we have enough for a header, process data... */
while (count = (int) sizeof(PipeProtoHeader))
***
*** 700,706
if (p.nuls[0] == '\0' p.nuls[1] == '\0'
p.len 0 p.len = PIPE_MAX_PAYLOAD
p.pid != 0
! (p.is_last == 't' || p.is_last == 'f'))
{
chunklen = PIPE_HEADER_SIZE + p.len;
--- 716,723
if (p.nuls[0] == '\0' p.nuls[1] == '\0'
p.len 0 p.len = PIPE_MAX_PAYLOAD
p.pid
Re: [PATCHES] Memory leak in nodeAgg
On Mon, 2007-08-06 at 14:21 -0700, Neil Conway wrote: Attached is a patch that fixes a gradual memory leak in ExecReScanAgg(), when the AGG_HASHED strategy is used Applied to HEAD, and backpatched back to 7.4. -Neil ---(end of broadcast)--- TIP 6: explain analyze is your friend
[PATCHES] Default location for docbook stylesheets in Gentoo
This patch adds the default location for the DocBook DSSSL stylesheets in gentoo's package system to the configure script. The package in question is app-text/docbook-dsssl-stylesheets. I'll understand if we don't want to include the location for every single distribution under the sun, but figured it's more help than harm to include it. Cheers BJ Index: configure === RCS file: /projects/cvsroot/pgsql/configure,v retrieving revision 1.556 diff -r1.556 configure 24263a24264 sgml/stylesheets/dsssl/docbook \ ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PATCHES] Default location for docbook stylesheets in Gentoo
On Thu, 2007-08-09 at 04:08 +1000, Brendan Jurd wrote: This patch adds the default location for the DocBook DSSSL stylesheets in gentoo's package system to the configure script. FYI, patches should typically be submitted as context diffs. For the specific case of configure, you should submit patches against the source file (configure.in), not the generated file (configure). -Neil ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Default location for docbook stylesheets in Gentoo
Brendan Jurd wrote: This patch adds the default location for the DocBook DSSSL stylesheets in gentoo's package system to the configure script. Uh, could you please send a diff with -u or -c, so we know where to put the change? -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] strpos() KMP
Tom Lane writes: I wonder why you didn't propose Boyer-Moore instead, as that would have some advantage for natural language text as well. The difficulty with B-M is the need for a table indexed by character code, which at first glance looks impractical for wchars. But it seems to me that we could use wchar % 256 as the table index, meaning that wchars with the same trailing byte share the same table entry. That would lose some efficiency compared to an exact implementation, but the limited table size would outweigh that except in the most pathological cases. hash table? The main difficulty with BM is verification and understanding good suffix shift (the second part of BM) (I don't understand it entirely). Ajtkulov Pavel [EMAIL PROTECTED] ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Default location for docbook stylesheets in Gentoo
On 8/9/07, Neil Conway [EMAIL PROTECTED] wrote: FYI, patches should typically be submitted as context diffs. For the specific case of configure, you should submit patches against the source file (configure.in), not the generated file (configure). Apologies. I haven't done much dev involving autoconf, so it didn't occur to me that configure wasn't a source. Is the following a reasonable way to test my changes? $ rm configure $ autoconf $ ./configure New (context) diff attached. The actual change is in config/docbook.m4, not configure.in. Cheers BJ Index: config/docbook.m4 === RCS file: /projects/cvsroot/pgsql/config/docbook.m4,v retrieving revision 1.7 diff -c -r1.7 docbook.m4 *** config/docbook.m4 13 Dec 2003 20:25:18 - 1.7 --- config/docbook.m4 8 Aug 2007 19:00:26 - *** *** 60,66 for pgac_postfix in \ sgml/stylesheets/nwalsh-modular \ sgml/stylesheets/docbook \ ! sgml/docbook-dsssl \ sgml/docbook/dsssl/modular \ sgml/docbook/stylesheet/dsssl/modular \ sgml/docbook/dsssl-stylesheets --- 60,67 for pgac_postfix in \ sgml/stylesheets/nwalsh-modular \ sgml/stylesheets/docbook \ ! sgml/stylesheets/dsssl/docbook \ ! sgml/docbook-dsssl \ sgml/docbook/dsssl/modular \ sgml/docbook/stylesheet/dsssl/modular \ sgml/docbook/dsssl-stylesheets ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] strpos() KMP
Pavel Ajtkulov [EMAIL PROTECTED] writes: Tom Lane writes: The difficulty with B-M is the need for a table indexed by character code, which at first glance looks impractical for wchars. But it seems to me that we could use wchar % 256 as the table index, meaning that wchars with the same trailing byte share the same table entry. hash table? I'd think the cost of hashing would render it impractical. Most of the papers I've seen on this topic worry about getting single instructions out of the search loop --- a hash lookup will cost lots more than that. Moreover, you'd lose the guarantee of not-worse-than-linear time, because hash lookup can be pathologically bad if you get a lot of hash collisions. The main difficulty with BM is verification and understanding good suffix shift (the second part of BM) (I don't understand it entirely). Yeah, there seem to be a bunch of variants of BM (many of them not guaranteed linear, which I'm sure we don't want) and the earliest papers had bugs. But a good implementation would be a lot easier sell because it would show benefits for a much wider set of use-cases than KMP. regards, tom lane ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PATCHES] Default location for docbook stylesheets in Gentoo
Brendan Jurd [EMAIL PROTECTED] writes: New (context) diff attached. The actual change is in config/docbook.m4, not configure.in. Much better. Committed in HEAD and REL8_2 (I suppose Gentoo weenies won't be interested in anything older ;-)) regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
