[GENERAL] SSL connection has been closed unexpectedly
Dear community We have a problem on our development database server, which supports a PHP application, which connects to it from a different server. Sometimes, around 1 in 4 page loads, it fails and reports the following error message: FATAL: terminating connection due to administrator command SSL connection has been closed unexpectedly Reloading the page usually works, sometimes doesn't, sometimes it requires several more refresh attempts before it magically works again. The odd thing is that we also have a live platform that is set up in the same way, and this does not occur, thankfully, but I expect it could. I've tried turning off all SSL features on the development platform, but oddly, the same problem persists. I've also tried whacking the logging level up to debug5, but still nothing appears in the PG logs when the problem occurs. Does anybody have any idea what could be happening here? Many thanks in advance Stuart Ford This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended only for the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or its content may result in legal liability. If you have received the email in error, please immediately notify us by telephone on +44 333 666 or email gl...@glide.uk.com The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of internet transmission. Any views/opinions expressed within this email and any attachments are that of the individual and not necessarily that of Glide Utilities Limited. Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway, Birmingham, B1 1TT. Registered in England Wales. Registered Company No. 06194523. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] SSL connection has been closed unexpectedly
Alban I would agree with you, except it still happens even after I have disabled all SSL related stuff in postgresql.conf and pg_hba.conf. I've also no evidence of any out of memory events on the server. Stuart -- From: Alban Hertroys haram...@gmail.com Date: Thursday, 15 August 2013 13:31 To: Stuart Ford stuart.f...@glide.uk.com Cc: pgsql-general@postgresql.org pgsql-general@postgresql.org Subject: Re: [GENERAL] SSL connection has been closed unexpectedly On 15 August 2013 12:41, Stuart Ford stuart.f...@glide.uk.com wrote: Dear community We have a problem on our development database server, which supports a PHP application, which connects to it from a different server. Sometimes, around 1 in 4 page loads, it fails and reports the following error message: FATAL: terminating connection due to administrator command SSL connection has been closed unexpectedly This has nothing to do with SSL. You have an administrator who's issuing commands to close database connections. Those just happen to be SSL connections. Perhaps the Linux OOM killer is at work here? -- If you can't see the forest for the trees, Cut the trees and you'll see there is no forest. This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended only for the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or its content may result in legal liability. If you have received the email in error, please immediately notify us by telephone on +44 333 666 or email gl...@glide.uk.com The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of internet transmission. Any views/opinions expressed within this email and any attachments are that of the individual and not necessarily that of Glide Utilities Limited. Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway, Birmingham, B1 1TT. Registered in England Wales. Registered Company No. 06194523. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] SSL connection has been closed unexpectedly
Guy No, we don't. It's also not happening on another platform which uses the same switch stack (and indeed VMWare cluster), so these aren't factors. Stuart On 15/08/2013 16:59, Guy Helmer ghel...@palisadesystems.com wrote: On Aug 15, 2013, at 5:41 AM, Stuart Ford stuart.f...@glide.uk.com wrote: Dear community We have a problem on our development database server, which supports a PHP application, which connects to it from a different server. Sometimes, around 1 in 4 page loads, it fails and reports the following error message: FATAL: terminating connection due to administrator command SSL connection has been closed unexpectedly Reloading the page usually works, sometimes doesn't, sometimes it requires several more refresh attempts before it magically works again. The odd thing is that we also have a live platform that is set up in the same way, and this does not occur, thankfully, but I expect it could. I've tried turning off all SSL features on the development platform, but oddly, the same problem persists. I've also tried whacking the logging level up to debug5, but still nothing appears in the PG logs when the problem occurs. Does anybody have any idea what could be happening here? Many thanks in advance Stuart Ford Any chance you are using HP ProCurve switches? I believe we have seen these switches corrupt SSL connections when systems use flow control signaling. Utterly bizarre behavior, but we've seen it at multiple customer sites. Guy This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended only for the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or its content may result in legal liability. If you have received the email in error, please immediately notify us by telephone on +44 333 666 or email gl...@glide.uk.com The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of internet transmission. Any views/opinions expressed within this email and any attachments are that of the individual and not necessarily that of Glide Utilities Limited. Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway, Birmingham, B1 1TT. Registered in England Wales. Registered Company No. 06194523. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
[GENERAL] soft lockup in kernel
Dear community Twice today our PG 9.1 server has caused a soft lockup, with a kernel message like this: [1813775.496127] BUG: soft lockup - CPU#3 stuck for 73s! [postgres:18723] Full dmesg output - http://pastebin.com/YdWSmNUp The incidents were approximately two hours apart and the server was momentarily unavailable before coming back again, with no restore actions required - it just carried on. It's never done this before, I've checked in the logs. The server is about 3 weeks old and runs Debian 7 under VMWare. Does anybody know what could cause this, and, if so, is it something to be concerned about and what can be done to stop it? Any help greatly appreciated. Stuart This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended only for the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or its content may result in legal liability. If you have received the email in error, please immediately notify us by telephone on +44 333 666 or email gl...@glide.uk.com The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of internet transmission. Any views/opinions expressed within this email and any attachments are that of the individual and not necessarily that of Glide Utilities Limited. Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway, Birmingham, B1 1TT. Registered in England Wales. Registered Company No. 06194523. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] soft lockup in kernel
On Fri, Jul 5, 2013 at 7:00 AM, Dennis Jenkins wrpte Before I looked at your pastebin, I was going to ask What kind of storage are the VMDKs on? If they are on NFS, iSCSI or FC, could the NAS/SAN be experiencing a problem? But I see in the stack trace that the kernel thread hung in vmxnet3_xmit_frame (sending an Ethernet frame on your virtual NIC). I did not spot that. Describe your vmware network topology. Do you share the same VLAN for guest traffic with NFS or iSCSI used by vmware for storing VMDKs? No. iSCSI traffic between the VMWare hosts and the SAN uses completely separate NICs and different switches to the production LAN. Are there any errors recorded on the Ethernet switch connected to your VMWare servers? No, there is nothing in the logs on the switch. What path does a packet take to get from a postgresql server process in your VM to a client? Postgres is the backend database to a web application. Only the web application uses it. It sits on another VM in the same cluster (but not on the same physical host, so it definitely goes through a physical switch). Is that what you mean? What version of VMWare are you running? ESXi 5.0.0. If you are managing it with vCenter, are there any alarms of events in the VMWare logs? I've had a look at the task activity in VCEnter and found these two events at almost the same time as the kernel messages. In both cases the start time (the first time below) is 5-6 seconds after the kernel message, and I've seen that the clock on the Postgres VM and the VCenter server, at least, are in sync (it may not, of course, be the VCenter server's clock that these logs get the time from). Remove snapshot GLIBMPDB001_replica Completed GLIDE\Svc_vcenter 05/07/2013 11:58:41 05/07/2013 11:58:41 05/07/2013 11:59:03 Remove snapshot GLIBMPDB001_replica Completed GLIDE\Svc_vcenter 05/07/2013 10:11:10 05/07/2013 10:11:10 05/07/2013 10:11:23 For clarity, here are the kernel messages: syslog:Jul 5 10:11:05 glibmpdb001 kernel: [1807328.506991] BUG: soft lockup - CPU#0 stuck for 25s! [postgres:5937] syslog:Jul 5 11:58:35 glibmpdb001 kernel: [1813775.496127] BUG: soft lockup - CPU#3 stuck for 73s! [postgres:18723] This is done by Veeam and not a person. There has not been a newer version of this event since (at the time of writing this e-mail). Regardless of this, it's not like we've only just introduced Veeam this morning, it's been running for a few weeks, and these lockups have only happened today. Do you think this is worth further investigation or could there be another explanation? Stuart This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended only for the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or its content may result in legal liability. If you have received the email in error, please immediately notify us by telephone on +44 333 666 or email gl...@glide.uk.com The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of internet transmission. Any views/opinions expressed within this email and any attachments are that of the individual and not necessarily that of Glide Utilities Limited. Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway, Birmingham, B1 1TT. Registered in England Wales. Registered Company No. 06194523. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pg_largeobject.sql script not run after upgrade
On 24/06/2013 19:20, Bruce Momjian br...@momjian.us wrote: On Mon, Jun 24, 2013 at 06:03:40PM +, Stuart Ford wrote: Do you know if not running this script would explain the fact that our dump file sizes have been much smaller than expected? It might be possible if lack of pg_largeobject_metadata values causes your large objects not to be dumped; I have not tested this. This did turn out to be the case in the end, FYI. Dump sizes returned to normal after the permissions were created. Many thanks for your assistance, much appreciated. Stuart This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended only for the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or its content may result in legal liability. If you have received the email in error, please immediately notify us by telephone on +44 333 666 or email gl...@glide.uk.com The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of internet transmission. Any views/opinions expressed within this email and any attachments are that of the individual and not necessarily that of Glide Utilities Limited. Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway, Birmingham, B1 1TT. Registered in England Wales. Registered Company No. 06194523. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
[GENERAL] pg_largeobject.sql script not run after upgrade
Dear community Last week we upgraded our database from 8.4 to 9.1. The upgrade seemed to go fine and the database seems to have been working fine ever since (around a week now). However, today I noticed the output from the pg_upgrade command contained the following: | Your installation contains large objects. | The new database has an additional large object | permission table so default permissions must be | defined for all large objects. The file: | /tmp/pg_largeobject.sql | when executed by psql by the database super-user | will define the default permissions. I missed this at the time, my fault, it was at the end of a stressful migration evening. Is it safe to run this script now, a week in to using the upgraded database? Can this be done while the database is live? Would appreciate your advice. Stuart Ford This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended only for the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or its content may result in legal liability. If you have received the email in error, please immediately notify us by telephone on +44 333 666 or email gl...@glide.uk.com The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of internet transmission. Any views/opinions expressed within this email and any attachments are that of the individual and not necessarily that of Glide Utilities Limited. Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway, Birmingham, B1 1TT. Registered in England Wales. Registered Company No. 06194523. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pg_largeobject.sql script not run after upgrade
On 24/06/2013 14:00, Bruce Momjian br...@momjian.us wrote: Looking further, here is the command that is executed: SELECT pg_catalog.lo_create(t.loid) FROM (SELECT DISTINCT loid FROM pg_catalog.pg_largeobject) AS t; If you have created _new_ large objects since the upgrde, the script might throw an error, as there is already metadata for those large objects. You might need to delete the rows in pg_largeobject_metadata before running the script; this will reset all the large object permissions to default. There doesn't appear to be, if this command, which returns 0, is correct: select count(*) from pg_catalog.pg_largeobject_metadata ; So it's OK to go ahead and run at any time? Stuart This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended only for the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or its content may result in legal liability. If you have received the email in error, please immediately notify us by telephone on +44 333 666 or email gl...@glide.uk.com The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of internet transmission. Any views/opinions expressed within this email and any attachments are that of the individual and not necessarily that of Glide Utilities Limited. Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway, Birmingham, B1 1TT. Registered in England Wales. Registered Company No. 06194523. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pg_largeobject.sql script not run after upgrade
On 24/06/2013 17:18, Bruce Momjian br...@momjian.us wrote: On Mon, Jun 24, 2013 at 03:25:44PM +, Stuart Ford wrote: On 24/06/2013 14:00, Bruce Momjian br...@momjian.us wrote: Looking further, here is the command that is executed: SELECT pg_catalog.lo_create(t.loid) FROM (SELECT DISTINCT loid FROM pg_catalog.pg_largeobject) AS t; If you have created _new_ large objects since the upgrde, the script might throw an error, as there is already metadata for those large objects. You might need to delete the rows in pg_largeobject_metadata before running the script; this will reset all the large object permissions to default. There doesn't appear to be, if this command, which returns 0, is correct: select count(*) from pg_catalog.pg_largeobject_metadata ; So it's OK to go ahead and run at any time? Yep. If it fails for some reason, just delete the contents of pg_largeobject_metadata and run it again. Do you know if not running this script would explain the fact that our dump file sizes have been much smaller than expected? Stuart This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended only for the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or its content may result in legal liability. If you have received the email in error, please immediately notify us by telephone on +44 333 666 or email gl...@glide.uk.com The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of internet transmission. Any views/opinions expressed within this email and any attachments are that of the individual and not necessarily that of Glide Utilities Limited. Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway, Birmingham, B1 1TT. Registered in England Wales. Registered Company No. 06194523. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
