Re: [Pgpool-general] unexpected EOF on client connection
On Wed, Sep 14, 2011 at 9:12 PM, Lonni J Friedman netll...@gmail.com wrote: On Wed, Sep 14, 2011 at 6:00 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: On Wed, Sep 14, 2011 at 4:22 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: I'm pretty sure that's not the case as the messages stop whenever pgpool isn't running, they were not present prior to using pgpool, and pg_hba.conf is setup such that the database servers only accept connections from each other, and the server running pgpool. None of these servers have normal users connected directly to them (such as with ssh), nor are they running anything that would connect to the database as a client. Also, the volume of these messages are such that something significant has to be causing them. Last night, in the span of 5 minutes, there were 117 of these messages. Ok. I would like to narraow down the reason why we have unexpected EOF on client connection message frequently. I think currently there are two possiblities: 1) pgpool child was killed by some unknown reason(we can omit segfault case because you don't see it in the pgpool log) 2) pgpool child disconnects to PostgreSQL in ungraceful manner For 1) I would like to know if pgpool child process are fine since they are spawned. Are you seeing any pgpool child process disappeared since pgpool started? I assume this should be determined by num_init_children (which I've set to 195 in pgpool.conf)? If so, then I currently have 195 processes in either the wait for connection request state or actively connected state. No. Pgpool parent process automatically respawns child process if it's dyning. So having num_init_children child process is not showing anything usefull. You record 195 process ids and compare current process ids. If some of them have been changed, we can assume that child process is dying. Ah, good point. I just diffed the list of PIDs associated with pgpool processes before and after another EOF message in the log, and there were no differences. So I think that rules out any processes dying? Right. One other thing that I just noticed from comparing logs between all of the database servers is that the time stamps for every one of the 'unexpected EOF on client connection' instances are identical. In other words, they are happening at the same time on each server. I think this further suggests that pgpool has to be doing it? Yes, I think so unless you set connection_life_time to other than 0 or the network connection between PostgreSQL and pgpool is unstable. I've used both pgpool and pgbouncer. We also have recurring similar client-EOF messages in both pgpool and pgbouncer logs. On pgbouncer, the actual error is: LOG C-0x7db270: mydb/myuser@10.0.0.160:43057 closing because: client unexpected eof (age=44005) On pgpool side, the error is exactly the same as Lonni's. So, from that, I had concluded that it was from our app side. But I have nothing further to substantiate that conclusion. Just adding it as another datapoint. Lonni, given how easy pgbouncer is to setup, it may be worth doing a quick proof-of-concept with it and see if you get similar EOF errors there. If so, I think that would eliminate the problem stemming from the middleware? johnny ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
On Thu, Sep 15, 2011 at 8:57 AM, Johnny Tan johnnyd...@gmail.com wrote: On Wed, Sep 14, 2011 at 9:12 PM, Lonni J Friedman netll...@gmail.com wrote: On Wed, Sep 14, 2011 at 6:00 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: On Wed, Sep 14, 2011 at 4:22 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: I'm pretty sure that's not the case as the messages stop whenever pgpool isn't running, they were not present prior to using pgpool, and pg_hba.conf is setup such that the database servers only accept connections from each other, and the server running pgpool. None of these servers have normal users connected directly to them (such as with ssh), nor are they running anything that would connect to the database as a client. Also, the volume of these messages are such that something significant has to be causing them. Last night, in the span of 5 minutes, there were 117 of these messages. Ok. I would like to narraow down the reason why we have unexpected EOF on client connection message frequently. I think currently there are two possiblities: 1) pgpool child was killed by some unknown reason(we can omit segfault case because you don't see it in the pgpool log) 2) pgpool child disconnects to PostgreSQL in ungraceful manner For 1) I would like to know if pgpool child process are fine since they are spawned. Are you seeing any pgpool child process disappeared since pgpool started? I assume this should be determined by num_init_children (which I've set to 195 in pgpool.conf)? If so, then I currently have 195 processes in either the wait for connection request state or actively connected state. No. Pgpool parent process automatically respawns child process if it's dyning. So having num_init_children child process is not showing anything usefull. You record 195 process ids and compare current process ids. If some of them have been changed, we can assume that child process is dying. Ah, good point. I just diffed the list of PIDs associated with pgpool processes before and after another EOF message in the log, and there were no differences. So I think that rules out any processes dying? Right. One other thing that I just noticed from comparing logs between all of the database servers is that the time stamps for every one of the 'unexpected EOF on client connection' instances are identical. In other words, they are happening at the same time on each server. I think this further suggests that pgpool has to be doing it? Yes, I think so unless you set connection_life_time to other than 0 or the network connection between PostgreSQL and pgpool is unstable. I've used both pgpool and pgbouncer. We also have recurring similar client-EOF messages in both pgpool and pgbouncer logs. On pgbouncer, the actual error is: LOG C-0x7db270: mydb/myuser@10.0.0.160:43057 closing because: client unexpected eof (age=44005) On pgpool side, the error is exactly the same as Lonni's. So, from that, I had concluded that it was from our app side. But I have nothing further to substantiate that conclusion. Just adding it as another datapoint. Lonni, given how easy pgbouncer is to setup, it may be worth doing a quick proof-of-concept with it and see if you get similar EOF errors there. If so, I think that would eliminate the problem stemming from the middleware? Actually I was using pgbouncer prior to switching to pgpool, and there were never any errors related to unexpected EOF. This definitely started since i switched to pgpool. ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
On Thu, Sep 15, 2011 at 11:58 AM, Lonni J Friedman netll...@gmail.com wrote: On Thu, Sep 15, 2011 at 8:57 AM, Johnny Tan johnnyd...@gmail.com wrote: Lonni, given how easy pgbouncer is to setup, it may be worth doing a quick proof-of-concept with it and see if you get similar EOF errors there. If so, I think that would eliminate the problem stemming from the middleware? Actually I was using pgbouncer prior to switching to pgpool, and there were never any errors related to unexpected EOF. This definitely started since i switched to pgpool. Ah interesting ok. Sorry for the noise, then. johnny ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
I'm pretty sure that's not the case as the messages stop whenever pgpool isn't running, they were not present prior to using pgpool, and pg_hba.conf is setup such that the database servers only accept connections from each other, and the server running pgpool. None of these servers have normal users connected directly to them (such as with ssh), nor are they running anything that would connect to the database as a client. Also, the volume of these messages are such that something significant has to be causing them. Last night, in the span of 5 minutes, there were 117 of these messages. Ok. I would like to narraow down the reason why we have unexpected EOF on client connection message frequently. I think currently there are two possiblities: 1) pgpool child was killed by some unknown reason(we can omit segfault case because you don't see it in the pgpool log) 2) pgpool child disconnects to PostgreSQL in ungraceful manner For 1) I would like to know if pgpool child process are fine since they are spawned. Are you seeing any pgpool child process disappeared since pgpool started? I assume this should be determined by num_init_children (which I've set to 195 in pgpool.conf)? If so, then I currently have 195 processes in either the wait for connection request state or actively connected state. No. Pgpool parent process automatically respawns child process if it's dyning. So having num_init_children child process is not showing anything usefull. You record 195 process ids and compare current process ids. If some of them have been changed, we can assume that child process is dying. -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
On Wed, Sep 14, 2011 at 4:22 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: I'm pretty sure that's not the case as the messages stop whenever pgpool isn't running, they were not present prior to using pgpool, and pg_hba.conf is setup such that the database servers only accept connections from each other, and the server running pgpool. None of these servers have normal users connected directly to them (such as with ssh), nor are they running anything that would connect to the database as a client. Also, the volume of these messages are such that something significant has to be causing them. Last night, in the span of 5 minutes, there were 117 of these messages. Ok. I would like to narraow down the reason why we have unexpected EOF on client connection message frequently. I think currently there are two possiblities: 1) pgpool child was killed by some unknown reason(we can omit segfault case because you don't see it in the pgpool log) 2) pgpool child disconnects to PostgreSQL in ungraceful manner For 1) I would like to know if pgpool child process are fine since they are spawned. Are you seeing any pgpool child process disappeared since pgpool started? I assume this should be determined by num_init_children (which I've set to 195 in pgpool.conf)? If so, then I currently have 195 processes in either the wait for connection request state or actively connected state. No. Pgpool parent process automatically respawns child process if it's dyning. So having num_init_children child process is not showing anything usefull. You record 195 process ids and compare current process ids. If some of them have been changed, we can assume that child process is dying. Ah, good point. I just diffed the list of PIDs associated with pgpool processes before and after another EOF message in the log, and there were no differences. So I think that rules out any processes dying? Right. One other thing that I just noticed from comparing logs between all of the database servers is that the time stamps for every one of the 'unexpected EOF on client connection' instances are identical. In other words, they are happening at the same time on each server. I think this further suggests that pgpool has to be doing it? Yes, I think so unless you set connection_life_time to other than 0 or the network connection between PostgreSQL and pgpool is unstable. Let me think how we can make further investigation... -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
On Wed, Sep 14, 2011 at 6:00 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: On Wed, Sep 14, 2011 at 4:22 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: I'm pretty sure that's not the case as the messages stop whenever pgpool isn't running, they were not present prior to using pgpool, and pg_hba.conf is setup such that the database servers only accept connections from each other, and the server running pgpool. None of these servers have normal users connected directly to them (such as with ssh), nor are they running anything that would connect to the database as a client. Also, the volume of these messages are such that something significant has to be causing them. Last night, in the span of 5 minutes, there were 117 of these messages. Ok. I would like to narraow down the reason why we have unexpected EOF on client connection message frequently. I think currently there are two possiblities: 1) pgpool child was killed by some unknown reason(we can omit segfault case because you don't see it in the pgpool log) 2) pgpool child disconnects to PostgreSQL in ungraceful manner For 1) I would like to know if pgpool child process are fine since they are spawned. Are you seeing any pgpool child process disappeared since pgpool started? I assume this should be determined by num_init_children (which I've set to 195 in pgpool.conf)? If so, then I currently have 195 processes in either the wait for connection request state or actively connected state. No. Pgpool parent process automatically respawns child process if it's dyning. So having num_init_children child process is not showing anything usefull. You record 195 process ids and compare current process ids. If some of them have been changed, we can assume that child process is dying. Ah, good point. I just diffed the list of PIDs associated with pgpool processes before and after another EOF message in the log, and there were no differences. So I think that rules out any processes dying? Right. One other thing that I just noticed from comparing logs between all of the database servers is that the time stamps for every one of the 'unexpected EOF on client connection' instances are identical. In other words, they are happening at the same time on each server. I think this further suggests that pgpool has to be doing it? Yes, I think so unless you set connection_life_time to other than 0 or the network connection between PostgreSQL and pgpool is unstable. connection _life_time is currently 0 (since you recommended I change it earlier). I don't have any evidence to suggest that the network connection is unstable. There are 0 errors of any kind in ifconfig output. Let me think how we can make further investigation... ok, thanks. ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
On Mon, Sep 12, 2011 at 6:47 PM, Lonni J Friedman netll...@gmail.com wrote: On Mon, Sep 12, 2011 at 6:39 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: I couldn't find anything possibly related to your problem at a first grance(in theory client_idle_limit and authentication_timeout are not related but you might want to change them to see anything could be changed). OK, I'll give that a try. Should I just try increasing them by 10 or 20s? I'd suggest giving them 0. This will prevent to initiate those functionalities which the directives are related. Also you hve child_life_time being 300. I don't expect this is related but could you set it to 0 and see anything gest changed for just in case? OK, i'll make those changes tomorrow (its late in the day here, and I don't want to introduce potential problems in the middle of the night when no one is closely monitoring the server), and let you know if they have any impact. client_idle_limit was already 0. I set authentication_timeout=0 and child_life_time=0, and restarted pgpool, however that had no impact. I'm still seeing: 26323 2011-09-13 09:28:19 PDT LOG: unexpected EOF on client connection 3933 2011-09-13 09:36:20 PDT LOG: unexpected EOF on client connection Do you have anything between pgpool and PostgreSQL? It has been reported that some firewall hardware/software kills TCP connections if they are idle for n seconds. Nope, there are no firewalls, or anything else that I'm aware of sitting between pgpool and the database servers. Ok. Another possibility is, pgpool child process is dying for unknown reason. Do you see anything bad (for example child dies segfault) in the pgpool log? I don't believe that's happening, but I'll have to check the logs tomorrow to verify. Currently, the only truly bad behavior that I'm currently experiencing is this: http://lists.pgfoundry.org/pipermail/pgpool-general/2011-September/003954.html I see no explicit segfaults in the pgpool logs. thanks for your help. ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
On Mon, Sep 12, 2011 at 6:47 PM, Lonni J Friedman netll...@gmail.com wrote: On Mon, Sep 12, 2011 at 6:39 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: I couldn't find anything possibly related to your problem at a first grance(in theory client_idle_limit and authentication_timeout are not related but you might want to change them to see anything could be changed). OK, I'll give that a try. Should I just try increasing them by 10 or 20s? I'd suggest giving them 0. This will prevent to initiate those functionalities which the directives are related. Also you hve child_life_time being 300. I don't expect this is related but could you set it to 0 and see anything gest changed for just in case? OK, i'll make those changes tomorrow (its late in the day here, and I don't want to introduce potential problems in the middle of the night when no one is closely monitoring the server), and let you know if they have any impact. client_idle_limit was already 0. I set authentication_timeout=0 and child_life_time=0, and restarted pgpool, however that had no impact. I'm still seeing: 26323 2011-09-13 09:28:19 PDT LOG: unexpected EOF on client connection 3933 2011-09-13 09:36:20 PDT LOG: unexpected EOF on client connection Humm. Is it possible that those connections do not come from pgpool process? -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp Do you have anything between pgpool and PostgreSQL? It has been reported that some firewall hardware/software kills TCP connections if they are idle for n seconds. Nope, there are no firewalls, or anything else that I'm aware of sitting between pgpool and the database servers. Ok. Another possibility is, pgpool child process is dying for unknown reason. Do you see anything bad (for example child dies segfault) in the pgpool log? I don't believe that's happening, but I'll have to check the logs tomorrow to verify. Currently, the only truly bad behavior that I'm currently experiencing is this: http://lists.pgfoundry.org/pipermail/pgpool-general/2011-September/003954.html I see no explicit segfaults in the pgpool logs. thanks for your help. ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
On Mon, Sep 12, 2011 at 3:34 AM, Toshihiro Kitagawa kitag...@sraoss.co.jp wrote: Hi, However, I'm see tons (dozens every minute) of the following in my postgresql server logs: LOG: unexpected EOF on client connection I found recently that it occurs if you specify the role that needs a password to health_check_user. Is your health_check_user trust authentication? Yes, all connections defined in pool_hba.conf are trust auth. However, I also have health_check_period = 0 in pgpool.conf, so I'd assume that no health checks are being performed? ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
Yes, all connections defined in pool_hba.conf are trust auth. However, I also have health_check_period = 0 in pgpool.conf, so I'd assume that no health checks are being performed? Have you changed child_life_time or something from defaults? I would like to take a look at your pgpool.conf. -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
On Mon, Sep 12, 2011 at 5:14 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: On Mon, Sep 12, 2011 at 4:16 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: Yes, all connections defined in pool_hba.conf are trust auth. However, I also have health_check_period = 0 in pgpool.conf, so I'd assume that no health checks are being performed? Have you changed child_life_time or something from defaults? I would like to take a look at your pgpool.conf. Nope, I'm using the default of 300. Anyway, pgpool.conf attached. Thanks for looking at this. I couldn't find anything possibly related to your problem at a first grance(in theory client_idle_limit and authentication_timeout are not related but you might want to change them to see anything could be changed). OK, I'll give that a try. Should I just try increasing them by 10 or 20s? Do you have anything between pgpool and PostgreSQL? It has been reported that some firewall hardware/software kills TCP connections if they are idle for n seconds. Nope, there are no firewalls, or anything else that I'm aware of sitting between pgpool and the database servers. ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
On Mon, Sep 12, 2011 at 6:39 PM, Tatsuo Ishii is...@sraoss.co.jp wrote: I couldn't find anything possibly related to your problem at a first grance(in theory client_idle_limit and authentication_timeout are not related but you might want to change them to see anything could be changed). OK, I'll give that a try. Should I just try increasing them by 10 or 20s? I'd suggest giving them 0. This will prevent to initiate those functionalities which the directives are related. Also you hve child_life_time being 300. I don't expect this is related but could you set it to 0 and see anything gest changed for just in case? OK, i'll make those changes tomorrow (its late in the day here, and I don't want to introduce potential problems in the middle of the night when no one is closely monitoring the server), and let you know if they have any impact. Do you have anything between pgpool and PostgreSQL? It has been reported that some firewall hardware/software kills TCP connections if they are idle for n seconds. Nope, there are no firewalls, or anything else that I'm aware of sitting between pgpool and the database servers. Ok. Another possibility is, pgpool child process is dying for unknown reason. Do you see anything bad (for example child dies segfault) in the pgpool log? I don't believe that's happening, but I'll have to check the logs tomorrow to verify. Currently, the only truly bad behavior that I'm currently experiencing is this: http://lists.pgfoundry.org/pipermail/pgpool-general/2011-September/003954.html ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
No one has any ideas or suggestions? On Fri, Sep 2, 2011 at 6:44 PM, Lonni J Friedman netll...@gmail.com wrote: Greetings, I just deployed a postgresql-9.0.4 cluster (1 master, 2 hot standby's), with pgpool-II-3.0.4 (all running on Linux-x86_64 servers). I'm currently only using pgpool for load balancing, and its working fine. However, I'm see tons (dozens every minute) of the following in my postgresql server logs: LOG: unexpected EOF on client connection This was not happening prior to setting up pgpool, so I'm currently working with the assumption that something in pgpool's load balancing might be causing it. Is this a known issue? If not, how can I debug this further? ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
Thanks for your reply. That doesn't appear to be the problem. While logged onto the server where pgpool is running, I can successfully invoke psql to connect directly to both of the standby servers (using port 5432). Also, I'm not seeing any errors in the pgpool log, where I'd expect to see something if connection attempts were failing? On Sun, Sep 4, 2011 at 4:43 PM, Richard Diekema rdiek...@gmail.com wrote: If I recall, unexpected EOF means your connection to one of the postgres instances is failing. Check your pg_hba.conf to make sure the pgpool has permission to connect, and that you can connect from the server running pgpool. On Sep 4, 2011 7:22 PM, Lonni J Friedman netll...@gmail.com wrote: No one has any ideas or suggestions? On Fri, Sep 2, 2011 at 6:44 PM, Lonni J Friedman netll...@gmail.com wrote: Greetings, I just deployed a postgresql-9.0.4 cluster (1 master, 2 hot standby's), with pgpool-II-3.0.4 (all running on Linux-x86_64 servers). I'm currently only using pgpool for load balancing, and its working fine. However, I'm see tons (dozens every minute) of the following in my postgresql server logs: LOG: unexpected EOF on client connection This was not happening prior to setting up pgpool, so I'm currently working with the assumption that something in pgpool's load balancing might be causing it. Is this a known issue? If not, how can I debug this further? ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
Re: [Pgpool-general] unexpected EOF on client connection
Unexpected EOF doesn't mean a postgres backend is failing; it typically means that a client has disconnected from its backend without closing out the session. (E.g., connect via psql, then kill the psql process -- not the backend! -- and you'll see an Unexpected EOF.) As for why you're only seeing them since starting to use pgpool, I couldn't say. I'm seeing them, too, but was before we started working with the pooler, too, and haven't noticed (but also haven't specifically looked for) a change in their frequency. rls [...] -- :wq ___ Pgpool-general mailing list Pgpool-general@pgfoundry.org http://pgfoundry.org/mailman/listinfo/pgpool-general
