Re: [Pdns-users] Backend error: Failed to execute mysql_query,perhaps connection died? Err=1: MySQL server has gone away
Again, this problem, for which a patch already exists, is triggered by the DB connection being terminated abnormally by the backend server, in this case via a connection time_limit. I do not use MySQL as a PDNS backend, but the PostgreSQL backend does correctly reconnect to the database should the connection die. I assume that the PDNS will reconnect to the MySQL server when the connection dies. It seems that increasing the timeout value in MySQL is that best measure to ensure more robust service from PDNS. Cheers, Ken On Tue, Jan 29, 2008 at 11:43:43AM -0800, Stephen Manchester wrote: > There is currently a memory leak in 2.9.21 from gmysql backend threads > dying in this fashion. The threads themselves are not being cleaned up. > this is only noticeable on very light loads where the backends do not get > enough traffic to keep the mysql connection open. You will notice it by > abnormally high virtual memory usage after pdns has been running for a > time. Each backend that dies leaves 10MB of virtual memory allocated (the > thread stack?) and this will not get cleaned up until pdns is restarted. > > Hopefully they will implement mysql connection reconnects in the future, > which solves both the memory leak and the backends failing when the > connection is closed. The only solution I found was to apply a patch and > compile from source. > > - Original Message - From: "Kenneth Marshall" <[EMAIL PROTECTED]> > To: "Catalin Constantin" <[EMAIL PROTECTED]> > Cc: > Sent: Tuesday, January 29, 2008 5:25 AM > Subject: Re: [Pdns-users] Backend error: Failed to execute > mysql_query,perhaps connection died? Err=1: MySQL server has gone away > > >> On Tue, Jan 29, 2008 at 03:04:46PM +0200, Catalin Constantin wrote: >>> Hello, >>> >>> This is a funny question. >>> Mysql is FINE and it's up and running on localhost. >>> >>> Here is the startup log of PDNS. >>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS 2.9.20 (C) 2001-2006 >>> PowerDNS.COMBV (Mar 10 2007, 00:36:58, gcc >>> 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) starting up >>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS comes with ABSOLUTELY NO >>> WARRANTY. >>> This is free software, and you are welcome to redistribute it according >>> to >>> the terms of the GPL version 2. >>> Jan 29 14:59:05 k2 pdns[30294]: Set effective group id to 109 >>> Jan 29 14:59:05 k2 pdns[30294]: Set effective user id to 113 >>> Jan 29 14:59:05 k2 pdns[30294]: Creating backend connection for TCP >>> Jan 29 14:59:05 k2 pdns[30294]: Master/slave communicator launching >>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful >>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful >>> Jan 29 14:59:05 k2 pdns[30294]: About to create 10 backend threads for >>> UDP >>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful >>> Jan 29 14:59:05 k2 pdns[30294]: 14 slave domains need checking >>> Jan 29 14:59:05 k2 pdns[30294]: Domain test1.com is stale, master serial >>> 1201611548, our serial 1201610827 >>> Jan 29 14:59:05 k2 pdns[30294]: Domain test2.com is stale, master serial >>> 1201611548, our serial 1201610827 >>> Jan 29 14:59:05 k2 pdns[30294]: Domain test3.com is stale, master serial >>> 1201611548, our serial 1201610827 >>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful >>> Jan 29 14:59:05 k2 pdns[30294]: AXFR started for 'test1.com', transaction >>> started >>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful >>> Jan 29 14:59:05 k2 pdns[30294]: AXFR done for 'test1.com', zone committed >>> >>> >>> The problem is that mysql closes the connection if there is no activity. >>> PDNS does not make sure that MySQL is still "alive" and from time to time >>> it >>> shows in the log errors like: >>> Jan 29 11:33:57 k2 pdns[28941]: Backend error: Failed to execute >>> mysql_query, perhaps connection died? Err=1: MySQL server has gone away >>> >>> Does PDNS have a timeout for backend connection "recheck" ? >>> Can this be configured ? >>> >>> Is there anything we can do to make sure the connection does not go away >>> ? >>> >>> Note: the NS server activity for now is not much (we're still testing and >>> just have a couple of domains on it). >>> >>> Thanks, >>> >> Dear Ms. Constantin, >> >> If a properly functioning application opens a connection to a database, >> t
Re: [Pdns-users] Backend error: Failed to execute mysql_query,perhaps connection died? Err=1: MySQL server has gone away
There is currently a memory leak in 2.9.21 from gmysql backend threads dying in this fashion. The threads themselves are not being cleaned up. this is only noticeable on very light loads where the backends do not get enough traffic to keep the mysql connection open. You will notice it by abnormally high virtual memory usage after pdns has been running for a time. Each backend that dies leaves 10MB of virtual memory allocated (the thread stack?) and this will not get cleaned up until pdns is restarted. Hopefully they will implement mysql connection reconnects in the future, which solves both the memory leak and the backends failing when the connection is closed. The only solution I found was to apply a patch and compile from source. - Original Message - From: "Kenneth Marshall" <[EMAIL PROTECTED]> To: "Catalin Constantin" <[EMAIL PROTECTED]> Cc: Sent: Tuesday, January 29, 2008 5:25 AM Subject: Re: [Pdns-users] Backend error: Failed to execute mysql_query,perhaps connection died? Err=1: MySQL server has gone away On Tue, Jan 29, 2008 at 03:04:46PM +0200, Catalin Constantin wrote: Hello, This is a funny question. Mysql is FINE and it's up and running on localhost. Here is the startup log of PDNS. Jan 29 14:59:05 k2 pdns[30294]: PowerDNS 2.9.20 (C) 2001-2006 PowerDNS.COMBV (Mar 10 2007, 00:36:58, gcc 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) starting up Jan 29 14:59:05 k2 pdns[30294]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2. Jan 29 14:59:05 k2 pdns[30294]: Set effective group id to 109 Jan 29 14:59:05 k2 pdns[30294]: Set effective user id to 113 Jan 29 14:59:05 k2 pdns[30294]: Creating backend connection for TCP Jan 29 14:59:05 k2 pdns[30294]: Master/slave communicator launching Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful Jan 29 14:59:05 k2 pdns[30294]: About to create 10 backend threads for UDP Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful Jan 29 14:59:05 k2 pdns[30294]: 14 slave domains need checking Jan 29 14:59:05 k2 pdns[30294]: Domain test1.com is stale, master serial 1201611548, our serial 1201610827 Jan 29 14:59:05 k2 pdns[30294]: Domain test2.com is stale, master serial 1201611548, our serial 1201610827 Jan 29 14:59:05 k2 pdns[30294]: Domain test3.com is stale, master serial 1201611548, our serial 1201610827 Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful Jan 29 14:59:05 k2 pdns[30294]: AXFR started for 'test1.com', transaction started Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful Jan 29 14:59:05 k2 pdns[30294]: AXFR done for 'test1.com', zone committed The problem is that mysql closes the connection if there is no activity. PDNS does not make sure that MySQL is still "alive" and from time to time it shows in the log errors like: Jan 29 11:33:57 k2 pdns[28941]: Backend error: Failed to execute mysql_query, perhaps connection died? Err=1: MySQL server has gone away Does PDNS have a timeout for backend connection "recheck" ? Can this be configured ? Is there anything we can do to make sure the connection does not go away ? Note: the NS server activity for now is not much (we're still testing and just have a couple of domains on it). Thanks, Dear Ms. Constantin, If a properly functioning application opens a connection to a database, the connection should not drop until closed. The typical fix is to increase your timeout on your MySQL server appropriately. This same thread comes up periodically. In order for PDNS to notice that a connection has died, it tries to use it with a query. This causes the errors in your logs. The amount of errors will be proportional to how long your timeout is set. I certainly would not like to see a patch that added overhead to every query to check the status of a connection and re-establish if needed. PDNS will re-connect if the connection drops, but you will get an error. If your MySQL DB is not dedicated to PDNS, then you will have to balance the needs of all of the applications using the DB. It sounds like you may be running some pretty poorly written ones. Good luck. Ken On 1/29/08, Nico van Royen <[EMAIL PROTECTED]> wrote: > Hello, > > Do you infact have a running mysql server ? Also check the database > that > you have specified in your pdns.conf matches your actual database > information (db name, user/password etc). > > Regards, > Nico > WARP3.COM > > From: Catalin Constantin [mailto:[EMAIL PROTECTED] > To: pdns-users@mailman.powerdns.com > Sent: Tue, 29 Jan 2008 10:48:09 +0100 > Subject: [Pdns-users] Backend error: Failed to execute mysql_query, perhaps > connection died? Err=1: MySQL server has gone away > >
Re: [Pdns-users] Backend error: Failed to execute mysql_query, perhaps connection died? Err=1: MySQL server has gone away
On Tue, Jan 29, 2008 at 03:04:46PM +0200, Catalin Constantin wrote: > Hello, > > This is a funny question. > Mysql is FINE and it's up and running on localhost. > > Here is the startup log of PDNS. > Jan 29 14:59:05 k2 pdns[30294]: PowerDNS 2.9.20 (C) 2001-2006 > PowerDNS.COMBV (Mar 10 2007, 00:36:58, gcc > 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) starting up > Jan 29 14:59:05 k2 pdns[30294]: PowerDNS comes with ABSOLUTELY NO WARRANTY. > This is free software, and you are welcome to redistribute it according to > the terms of the GPL version 2. > Jan 29 14:59:05 k2 pdns[30294]: Set effective group id to 109 > Jan 29 14:59:05 k2 pdns[30294]: Set effective user id to 113 > Jan 29 14:59:05 k2 pdns[30294]: Creating backend connection for TCP > Jan 29 14:59:05 k2 pdns[30294]: Master/slave communicator launching > Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful > Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful > Jan 29 14:59:05 k2 pdns[30294]: About to create 10 backend threads for UDP > Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful > Jan 29 14:59:05 k2 pdns[30294]: 14 slave domains need checking > Jan 29 14:59:05 k2 pdns[30294]: Domain test1.com is stale, master serial > 1201611548, our serial 1201610827 > Jan 29 14:59:05 k2 pdns[30294]: Domain test2.com is stale, master serial > 1201611548, our serial 1201610827 > Jan 29 14:59:05 k2 pdns[30294]: Domain test3.com is stale, master serial > 1201611548, our serial 1201610827 > Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful > Jan 29 14:59:05 k2 pdns[30294]: AXFR started for 'test1.com', transaction > started > Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful > Jan 29 14:59:05 k2 pdns[30294]: AXFR done for 'test1.com', zone committed > > > The problem is that mysql closes the connection if there is no activity. > PDNS does not make sure that MySQL is still "alive" and from time to time it > shows in the log errors like: > Jan 29 11:33:57 k2 pdns[28941]: Backend error: Failed to execute > mysql_query, perhaps connection died? Err=1: MySQL server has gone away > > Does PDNS have a timeout for backend connection "recheck" ? > Can this be configured ? > > Is there anything we can do to make sure the connection does not go away ? > > Note: the NS server activity for now is not much (we're still testing and > just have a couple of domains on it). > > Thanks, > Dear Ms. Constantin, If a properly functioning application opens a connection to a database, the connection should not drop until closed. The typical fix is to increase your timeout on your MySQL server appropriately. This same thread comes up periodically. In order for PDNS to notice that a connection has died, it tries to use it with a query. This causes the errors in your logs. The amount of errors will be proportional to how long your timeout is set. I certainly would not like to see a patch that added overhead to every query to check the status of a connection and re-establish if needed. PDNS will re-connect if the connection drops, but you will get an error. If your MySQL DB is not dedicated to PDNS, then you will have to balance the needs of all of the applications using the DB. It sounds like you may be running some pretty poorly written ones. Good luck. Ken > On 1/29/08, Nico van Royen <[EMAIL PROTECTED]> wrote: > > Hello, > > > > Do you infact have a running mysql server ? Also check the database that > > you have specified in your pdns.conf matches your actual database > > information (db name, user/password etc). > > > > Regards, > > Nico > > WARP3.COM > > > > From: Catalin Constantin [mailto:[EMAIL PROTECTED] > > To: pdns-users@mailman.powerdns.com > > Sent: Tue, 29 Jan 2008 10:48:09 +0100 > > Subject: [Pdns-users] Backend error: Failed to execute mysql_query, > perhaps > > connection died? Err=1: MySQL server has gone away > > > > > > Hello, > > > > We're getting quite many errors: > > Backend error: Failed to execute mysql_query, perhaps connection died? > > Err=1: MySQL server has gone away latelly. > > > > I checked the docs and the mailing lists for this error and i found > > little informations about it. > > > > What is actually the solution for this error not to happen ? > > Nice would be to have a real example. > > > > Note: > > - we use latest pdns package from Debian Etch (stable). > > > > Thanks, > > > > -- > > Catalin Constantin > > Dazoot Software > > http://www.dazoot.eu/ > > ___ > > Pdns-users mailing list > > Pdns-users@mailman.powerdns.com > > http://mailman.powerdns.com/mailman/listinfo/pdns-users > > > > ___ > > Pdns-users mailing list > > Pdns-users@mailman.powerdns.com > > http://mailman.powerdns.com/mailman/listinfo/pdns-users > > > > > > > -- > Catalin Constantin > Dazoot Software > http://www.dazoot.eu/ > ___ > Pdns-users mailing lis
Re: [Pdns-users] Backend error: Failed to execute mysql_query, perhaps connection died? Err=1: MySQL server has gone away
Hello, This is a funny question. Mysql is FINE and it's up and running on localhost. Here is the startup log of PDNS. Jan 29 14:59:05 k2 pdns[30294]: PowerDNS 2.9.20 (C) 2001-2006 PowerDNS.COMBV (Mar 10 2007, 00:36:58, gcc 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) starting up Jan 29 14:59:05 k2 pdns[30294]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2. Jan 29 14:59:05 k2 pdns[30294]: Set effective group id to 109 Jan 29 14:59:05 k2 pdns[30294]: Set effective user id to 113 Jan 29 14:59:05 k2 pdns[30294]: Creating backend connection for TCP Jan 29 14:59:05 k2 pdns[30294]: Master/slave communicator launching Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful Jan 29 14:59:05 k2 pdns[30294]: About to create 10 backend threads for UDP Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful Jan 29 14:59:05 k2 pdns[30294]: 14 slave domains need checking Jan 29 14:59:05 k2 pdns[30294]: Domain test1.com is stale, master serial 1201611548, our serial 1201610827 Jan 29 14:59:05 k2 pdns[30294]: Domain test2.com is stale, master serial 1201611548, our serial 1201610827 Jan 29 14:59:05 k2 pdns[30294]: Domain test3.com is stale, master serial 1201611548, our serial 1201610827 Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful Jan 29 14:59:05 k2 pdns[30294]: AXFR started for 'test1.com', transaction started Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful Jan 29 14:59:05 k2 pdns[30294]: AXFR done for 'test1.com', zone committed The problem is that mysql closes the connection if there is no activity. PDNS does not make sure that MySQL is still "alive" and from time to time it shows in the log errors like: Jan 29 11:33:57 k2 pdns[28941]: Backend error: Failed to execute mysql_query, perhaps connection died? Err=1: MySQL server has gone away Does PDNS have a timeout for backend connection "recheck" ? Can this be configured ? Is there anything we can do to make sure the connection does not go away ? Note: the NS server activity for now is not much (we're still testing and just have a couple of domains on it). Thanks, On 1/29/08, Nico van Royen <[EMAIL PROTECTED]> wrote: > Hello, > > Do you infact have a running mysql server ? Also check the database that > you have specified in your pdns.conf matches your actual database > information (db name, user/password etc). > > Regards, > Nico > WARP3.COM > > From: Catalin Constantin [mailto:[EMAIL PROTECTED] > To: pdns-users@mailman.powerdns.com > Sent: Tue, 29 Jan 2008 10:48:09 +0100 > Subject: [Pdns-users] Backend error: Failed to execute mysql_query, perhaps > connection died? Err=1: MySQL server has gone away > > > Hello, > > We're getting quite many errors: > Backend error: Failed to execute mysql_query, perhaps connection died? > Err=1: MySQL server has gone away latelly. > > I checked the docs and the mailing lists for this error and i found > little informations about it. > > What is actually the solution for this error not to happen ? > Nice would be to have a real example. > > Note: > - we use latest pdns package from Debian Etch (stable). > > Thanks, > > -- > Catalin Constantin > Dazoot Software > http://www.dazoot.eu/ > ___ > Pdns-users mailing list > Pdns-users@mailman.powerdns.com > http://mailman.powerdns.com/mailman/listinfo/pdns-users > > ___ > Pdns-users mailing list > Pdns-users@mailman.powerdns.com > http://mailman.powerdns.com/mailman/listinfo/pdns-users > > -- Catalin Constantin Dazoot Software http://www.dazoot.eu/ ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Backend error: Failed to execute mysql_query, perhaps connection died ? Err=1: MySQL server has gone away
Hello, Do you infact have a running mysql server ? Also check the database that you have specified in your pdns.conf matches your actual database information (db name, user/password etc). Regards, Nico WARP3.COM _ From: Catalin Constantin [mailto:[EMAIL PROTECTED] To: pdns-users@mailman.powerdns.com Sent: Tue, 29 Jan 2008 10:48:09 +0100 Subject: [Pdns-users] Backend error: Failed to execute mysql_query, perhaps connection died? Err=1: MySQL server has gone away Hello, We're getting quite many errors: Backend error: Failed to execute mysql_query, perhaps connection died? Err=1: MySQL server has gone away latelly. I checked the docs and the mailing lists for this error and i found little informations about it. What is actually the solution for this error not to happen ? Nice would be to have a real example. Note: - we use latest pdns package from Debian Etch (stable). Thanks, -- Catalin Constantin Dazoot Software http://www.dazoot.eu/ ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
