Early this morning one of our databases suffered a segfault. The only note on this in /var/log/messages was:
Nov 29 03:34:07 servername kernel: postmaster[18265]: segfault at 0000000017b82000 rip 0000003c45e7c266 rsp 00007fff2ca1b7d8 error 4 (replaced actual server name with servername) The postgres log has a bit more information: 2010-11-29 03:34:07 PST LOG: server process (PID 18265) was terminated by signal 11 2010-11-29 03:34:07 PST LOG: terminating any other active server processes 2010-11-29 03:34:07 PST 10.2.0.18 WARNING: terminating connection because of crash of another server process 2010-11-29 03:34:07 PST 10.2.0.18 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2010-11-29 03:34:07 PST 10.2.0.18 HINT: In a moment you should be able to reconnect to the database and repeat your command. 2010-11-29 03:34:07 PST 10.2.0.18 WARNING: terminating connection because of crash of another server process 2010-11-29 03:34:07 PST 10.2.0.18 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2010-11-29 03:34:07 PST 10.2.0.18 HINT: In a moment you should be able to reconnect to the database and repeat your command. 2010-11-29 03:34:07 PST 10.2.0.18 WARNING: terminating connection because of crash of another server process 2010-11-29 03:34:07 PST 10.2.0.18 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2010-11-29 03:34:07 PST 10.2.0.18 HINT: In a moment you should be able to reconnect to the database and repeat your command. 2010-11-29 03:34:07 PST 10.2.0.18 FATAL: the database system is in recovery mode 2010-11-29 03:34:07 PST LOG: all server processes terminated; reinitializing 2010-11-29 03:34:10 PST 10.2.0.17 FATAL: the database system is starting up 2010-11-29 03:34:10 PST 10.2.0.17 FATAL: the database system is starting up 2010-11-29 03:34:10 PST 127.0.0.1 FATAL: the database system is starting up 2010-11-29 03:34:10 PST 10.2.0.17 FATAL: the database system is starting up 2010-11-29 03:34:10 PST 10.2.0.18 FATAL: the database system is starting up 2010-11-29 03:34:10 PST 10.2.0.18 FATAL: the database system is starting up 2010-11-29 03:34:10 PST LOG: database system was interrupted at 2010-11-29 03:30:26 PST 2010-11-29 03:34:10 PST LOG: checkpoint record is at 41F/723A1648 2010-11-29 03:34:10 PST LOG: redo record is at 41F/723A1648; undo record is at 0/0; shutdown FALSE 2010-11-29 03:34:10 PST LOG: next transaction ID: 0/1144655884; next OID: 630827 2010-11-29 03:34:10 PST LOG: next MultiXactId: 1; next MultiXactOffset: 0 2010-11-29 03:34:10 PST LOG: database system was not properly shut down; automatic recovery in progress 2010-11-29 03:34:10 PST LOG: record with zero length at 41F/723A1698 2010-11-29 03:34:10 PST LOG: redo is not required 2010-11-29 03:34:10 PST 10.2.0.17 FATAL: the database system is starting up 2010-11-29 03:34:10 PST LOG: database system is ready I’m trying to figure out what might have caused this, partly because I want to replicate the segfault to see if it is the cause of another system randomly freezing on us. Is there any way to figure out exactly what query being run at the time of the crash was doing? Could the xlog files help me in anyway?
