Re: mysql error 2013 Lost connection to MySQL server during query
Per Jessen wrote: It happened agaIn this morning, but slightly different: [snip] thd=0x7fe0140c7e00 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xb, backtrace may not be correct. Bogus stack limit or frame pointer, fp=0xb, stack_bottom=0x4514, thread_stack=262144, aborting backtrace. Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd-query at 0x1355140 = INSERT IGNORE INTO quarantine_archive SELECT * FROM quarantine WHERE state=1 AND domain='example.com' thd-thread_id=1493537 The context is the same as previously, except the query: INSERT IGNORE INTO quarantine_archive SELECT * FROM quarantine WHERE state=1 AND domain='example.com' This is not exactly reproducable, but it is fairly predictable - happens every morning towards 0600 - I have an archive job starting at 0500. For the last three days, the query has been roughly the same, except the 'example.com' varies. Is there nothing I can do to attempt to diagnose crashes such as this? Still no suggestions? /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Per Jessen wrote: I have just discovered that my mysql server was restarted this morning, which is what gave me the 2013. In the log I found this: [snip] It happened agaIn this morning, but slightly different: [snip] thd=0x7fe0140c7e00 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xb, backtrace may not be correct. Bogus stack limit or frame pointer, fp=0xb, stack_bottom=0x4514, thread_stack=262144, aborting backtrace. Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd-query at 0x1355140 = INSERT IGNORE INTO quarantine_archive SELECT * FROM quarantine WHERE state=1 AND domain='example.com' thd-thread_id=1493537 The context is the same as previously, except the query: INSERT IGNORE INTO quarantine_archive SELECT * FROM quarantine WHERE state=1 AND domain='example.com' It's getting to be a bit annoying - not all our apps were written to be able to handle the database connection disappearing at any time. Yes, they should have been, but it is a pretty unusual situation after all. Is there nothing I can do to attempt to diagnose crashes such as this? /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
mysql error 2013 Lost connection to MySQL server during query
This weekend we completed migrating a large(ish) mysql server from 5.0.26 on 32bit to 5.0.51a on 64bit. Everything went relatively smoothly, until this morning when I noticed an application had choked on getting Error 2013 Lost connection to MySQL server during query. The application is running remotely on 32bit using mysql library from version 5.0.67. I've been googling quite a bit, but haven't really found anything of any use. I've checked the two configurations, and they are the same. Can anyone help point me in the right direction? Thanks. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Per Jessen wrote: This weekend we completed migrating a large(ish) mysql server from 5.0.26 on 32bit to 5.0.51a on 64bit. Everything went relatively smoothly, until this morning when I noticed an application had choked on getting Error 2013 Lost connection to MySQL server during query. I have just discovered that my mysql server was restarted this morning, which is what gave me the 2013. In the log I found this: 090525 6:04:35 - mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=6442450944 read_buffer_size=258048 max_used_connections=43 max_connections=100 threads_connected=26 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 6367855 Kbytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x7fa6fc0173e0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xb, backtrace may not be correct. Bogus stack limit or frame pointer, fp=0xb, stack_bottom=0x41a6, thread_stack=262144, aborting backtrace. Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd-query at 0x1340aa0 = SELECT domain,domain FROM dodgy_domain,spamdns_ipaddr WHERE dodgy_domain.ipaddr=spamdns_ipaddr.ipaddr group by domain having min(first)='2009-05-25 00:00:00' thd-thread_id=434983 /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Michael Dykman wrote: It might be helpful if you could tell us how you affected your data migration Sorry, I'm not familiar with reporting problems in/on mysql. The data migration was done with a full database dump (mysqldump) from the 32bit system, then a reload on the new 64bit system. I think it took 6-8 hours. and what kind of job was running at the time it went down. The job executing the SQL mentioned in the log ran on another server. It is a SELECT running from the command line (in a Makefile). I'm not sure what else to tell you. Having the server go away mid-query generally does mean you have run into a bug of some sort but, more often than not, you were doing something ill-advised at the time. The setup has been running for at least two years with no such problems. Let me know what other info would be interesting. best regards Per Jessen -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
just a thought: Did you run mysql_upgrade after the import? On Mon, May 25, 2009 at 10:19 AM, Per Jessen p...@computer.org wrote: Michael Dykman wrote: It might be helpful if you could tell us how you affected your data migration Sorry, I'm not familiar with reporting problems in/on mysql. The data migration was done with a full database dump (mysqldump) from the 32bit system, then a reload on the new 64bit system. I think it took 6-8 hours. and what kind of job was running at the time it went down. The job executing the SQL mentioned in the log ran on another server. It is a SELECT running from the command line (in a Makefile). I'm not sure what else to tell you. Having the server go away mid-query generally does mean you have run into a bug of some sort but, more often than not, you were doing something ill-advised at the time. The setup has been running for at least two years with no such problems. Let me know what other info would be interesting. best regards Per Jessen -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql?unsub=mikeisgr...@gmail.com -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Michael Steinfeld wrote: just a thought: Did you run mysql_upgrade after the import? No, I didn't - I didn't think of it as I really only moved the data across. best regards Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Per Jessen wrote: Michael Steinfeld wrote: just a thought: Did you run mysql_upgrade after the import? No, I didn't - I didn't think of it as I really only moved the data across. Okay, have done a mysqlcheck --check-upgrade - came back all clean. I don't see a need to run mysql_fix_privilege as I manually copied the necessary privilege data. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Have you tried running the offending SQL manually against you new installation? Does it come back clean in the isolated case? Is there anything else which runs against this database at night? crons? Could you post the script that you are running to give some context to the statement which winds up in your error log? Considering that you did use mysqldump to manually inject your data, cross-version incompatibilities are pretty much out of the question. - michael On Mon, May 25, 2009 at 11:19 AM, Per Jessen p...@computer.org wrote: Per Jessen wrote: Michael Steinfeld wrote: just a thought: Did you run mysql_upgrade after the import? No, I didn't - I didn't think of it as I really only moved the data across. Okay, have done a mysqlcheck --check-upgrade - came back all clean. I don't see a need to run mysql_fix_privilege as I manually copied the necessary privilege data. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql?unsub=mdyk...@gmail.com -- - michael dykman - mdyk...@gmail.com - All models are wrong. Some models are useful. -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
On Mon, May 25, 2009 at 11:19 AM, Per Jessen p...@computer.org wrote: Per Jessen wrote: Michael Steinfeld wrote: just a thought: Did you run mysql_upgrade after the import? No, I didn't - I didn't think of it as I really only moved the data across. I suspect that will solve your issue. Keep me posted. Okay, have done a mysqlcheck --check-upgrade - came back all clean. I don't see a need to run mysql_fix_privilege as I manually copied the necessary privilege data. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql?unsub=mikeisgr...@gmail.com -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Michael Dykman wrote: Have you tried running the offending SQL manually against you new installation? Does it come back clean in the isolated case? No, not manually, but the job/the SQL is run several times a day, maybe 2-3 times per hour. Is there anything else which runs against this database at night? crons? Yes, lots of stuff. Cron-jobs, jobs submitted by daemons, etc. Could you post the script that you are running to give some context to the statement which winds up in your error log? I'm generating a zonefile for rbldnsd with entries from my table since midnight. Entries from before midnight are put in a main-table, entries after are in this regular diff. The statement is this: SELECT domain,domain FROM dodgy_domain,spamdns_ipaddr WHERE dodgy_domain.ipaddr=spamdns_ipaddr.ipaddr group by domain having min(first)='midnight' Tonight it will be changed to: (single domain, not domain,domain) SELECT domain FROM dodgy_domain,spamdns_ipaddr WHERE dodgy_domain.ipaddr=spamdns_ipaddr.ipaddr group by domain having min(first)='midnight' Additional context: The server is brandnew, an HP Proliant with dual quad-core Xeons and 10Gb RAM. The filesystem is JFS on hardware RAID6. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Per Jessen wrote: Michael Dykman wrote: Have you tried running the offending SQL manually against you new installation? Does it come back clean in the isolated case? No, not manually, but the job/the SQL is run several times a day, maybe 2-3 times per hour. I've also just run the query manually a couple of times, no problems. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Given the new hardware, I'm now suspecting the RAID controller. I have seen misconfigured RAIDs or bad RAID drivers take out a server in just such a manner. I had a debian server connected to an EMC SAN.. As debian isn't supported, we had this open-source driver which gave us no end of problems. If a logical drive acts up or does something unexpected, MySQL could react to that in a manner consistent with what you are seeing in your log. I would be tempted to put the hardware through a stress test. I know that's not much help. - michael On Mon, May 25, 2009 at 12:02 PM, Per Jessen p...@computer.org wrote: Per Jessen wrote: Michael Dykman wrote: Have you tried running the offending SQL manually against you new installation? Does it come back clean in the isolated case? No, not manually, but the job/the SQL is run several times a day, maybe 2-3 times per hour. I've also just run the query manually a couple of times, no problems. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql?unsub=mdyk...@gmail.com -- - michael dykman - mdyk...@gmail.com - All models are wrong. Some models are useful. -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Michael Dykman wrote: Given the new hardware, I'm now suspecting the RAID controller. I have seen misconfigured RAIDs or bad RAID drivers take out a server in just such a manner. I had a debian server connected to an EMC SAN.. As debian isn't supported, we had this open-source driver which gave us no end of problems. If a logical drive acts up or does something unexpected, MySQL could react to that in a manner consistent with what you are seeing in your log. Shouldn't/wouldn't the filesystem complain first? There is a lot of activity on the filesystem, mysql is just a tiny part of it. I would be tempted to put the hardware through a stress test. I know that's not much help. I really have no reason to suspect the hardware. It's new, but it's been running in burn-in mode for about a month (although not with much load, mostly idling). I might as well suspect the mysql build and try upgrading to a newer one. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
The issues that we saw only came to light under stress. The application I am referring to ran under a fair bit of load at the best of times but it was during sustained spikes that the flaws in our driver made themselves apparent. Mind you, we weren't using JFS, so I'm not sure how that would have reacted. - michael On Mon, May 25, 2009 at 12:19 PM, Per Jessen p...@computer.org wrote: Michael Dykman wrote: Given the new hardware, I'm now suspecting the RAID controller. I have seen misconfigured RAIDs or bad RAID drivers take out a server in just such a manner. I had a debian server connected to an EMC SAN.. As debian isn't supported, we had this open-source driver which gave us no end of problems. If a logical drive acts up or does something unexpected, MySQL could react to that in a manner consistent with what you are seeing in your log. Shouldn't/wouldn't the filesystem complain first? There is a lot of activity on the filesystem, mysql is just a tiny part of it. I would be tempted to put the hardware through a stress test. I know that's not much help. I really have no reason to suspect the hardware. It's new, but it's been running in burn-in mode for about a month (although not with much load, mostly idling). I might as well suspect the mysql build and try upgrading to a newer one. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql?unsub=mdyk...@gmail.com -- - michael dykman - mdyk...@gmail.com - All models are wrong. Some models are useful. -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Hi Per, Maybe you need to beef up your CONNECT_TIMEOUT setting in your .my.cnf file. Are these queries appearing in your slow query logs?What is your LOG_QUERY_TIMES set too? Here are some other settings you may want to play around wtih CONNECT_TIMEOUT INTERACTIVE_TIMEOUT WAIT_TIMEOUT NET_WRITE_TIMEOUT NET_READ_TIMEOUT MAX_CONNECT_ERRORS On Mon, May 25, 2009 at 3:06 AM, Per Jessen p...@computer.org wrote: This weekend we completed migrating a large(ish) mysql server from 5.0.26 on 32bit to 5.0.51a on 64bit. Everything went relatively smoothly, until this morning when I noticed an application had choked on getting Error 2013 Lost connection to MySQL server during query. The application is running remotely on 32bit using mysql library from version 5.0.67. I've been googling quite a bit, but haven't really found anything of any use. I've checked the two configurations, and they are the same. Can anyone help point me in the right direction? Thanks. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql?unsub=dstepli...@gmail.com -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
Re: mysql error 2013 Lost connection to MySQL server during query
Darryle Steplight wrote: Hi Per, Maybe you need to beef up your CONNECT_TIMEOUT setting in your .my.cnf file. Are these queries appearing in your slow query logs?What is your LOG_QUERY_TIMES set too? Here are some other settings you may want to play around wtih CONNECT_TIMEOUT INTERACTIVE_TIMEOUT WAIT_TIMEOUT NET_WRITE_TIMEOUT NET_READ_TIMEOUT MAX_CONNECT_ERRORS Hi Darryle I did notice references to some of those when I was googling, but because I didn't change any settings in my migration except up the key_buffer space, I didn't really pay much attention. /Per Jessen, Zürich -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql?unsub=arch...@jab.org
