memory error innodb backup
We've got a customer whose system has been experiencing corruption in their InnoDB tables. They have returned the system to us and after testing we've determined that the server has bad RAM. This is the same system I had posted about in http://lists.mysql.com/mysql/180785 Memtest86+ v1.55 can run anywhere from 12-24 hours before it picks up any memory errors, so the problem with the memory (or memory subsystem) is very intermittent. We have shipped this customer a replacement server and they have restored their database from a backup that was made with the InnoDB hot backup utility (ibbackup, 1.40). They are now experiencing corruption in their database again on the new system. My question is this: Is it possible that the backup made from the system with the failing RAM is itself corrupted somehow? If there was a corrupt page in the tablespace, would it generate an error during the ibbackup run? Would the restore process be able to uncover an error within the backup? Or is it possible that a corrupted page could survive through the backup/restore process and not be uncovered until it was accessed on the new system? thanks, baba -- Baba Buehler - NetBotz, Inc. - [EMAIL PROTECTED] -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: mysql 4.0.18 crashing on startup
Heikki Tuuri wrote: Baba, an undo log seems to be corrupt. He should restart mysqld with: innodb_force_recovery=4 dump all tables, and recreate the whole InnoDB installation. This was roughly my assessment and recommended procedure as well. The customer decided to restore the system from a full backup they had made about a week earlier. Did the .err log contain any hint what could be behind the corruption? No, the system had been up for a couple of months (since they last patched some other software) and the .err file didn't report any abnormalities up until MySQL was restarted and the stack traces began. We scanned the disks on the system for bad blocks and found none, and the system had been on UPS power and had not suffered any harsh reboots. We have many hundreds of MySQL/InnoDB servers in the field, and occationally see db corruption. Most of the time we trace it back to faulty disk hardware, but occationally we see one like this where we can't find the smoking gun. Is there any setting or anything you recommend turning on a system that is suspect to divine more information on where something like this may have originated? thanks, baba Best regards, Heikki Tuuri Innobase Oy Foreign keys, transactions, and row level locking for MySQL InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up MyISAM tables http://www.innodb.com/order.php - Original Message - From: Baba Buehler [EMAIL PROTECTED] Newsgroups: mailing.database.myodbc Sent: Wednesday, March 02, 2005 6:32 PM Subject: mysql 4.0.18 crashing on startup I've got a customer with a mysql that is crashing on startup. MySQL is 4.0.18 with InnoDB tables on a Linux 2.4.26 system. MySQL binaries are Linux x86 glibc static gcc from mysql.com Can anyone divine what might have happened to cause this? thanks, baba resolve_stack_dump gives: 0x8071f44 handle_segfault + 420 0x82a0e38 pthread_sighandler + 184 0x818cfeb trx_undo_rec_get_partial_row + 443 0x81302af row_purge_parse_undo_rec + 863 0x813033f row_purge + 127 0x81304d7 row_purge_step + 23 0x8115aa3 que_run_threads + 755 0x8189520 trx_purge + 2304 0x811b8dd srv_master_thread + 1389 0x829e5ec pthread_start_thread + 220 0x82c7dea thread_start + 4 the .err log shows this, ever repeating: 050222 10:49:57 mysqld started 050222 10:49:57 InnoDB: Started /usr/sbin/mysqld: ready for connections. Version: '4.0.18-standard' socket: '/tmp/mysql4.sock' port: 3306 mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=1048576 read_buffer_size=4190208 max_used_connections=0 max_connections=35 threads_connected=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 287603 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=(nil) Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xbfebf5b8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x8071f44 0x82a0e38 0x818cfc1 0x81302af 0x813033f 0x81304d7 0x8115aa3 0x8189520 0x811bb6f 0x829e5ec 0x82c7dea New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 1 050222 10:49:58 mysqld restarted 050222 10:49:59 InnoDB: Started /usr/sbin/mysqld: ready for connections. Version: '4.0.18-standard' socket: '/tmp/mysql4.sock' port: 3306 mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=1048576 read_buffer_size=4190208 max_used_connections=0 max_connections=35 threads_connected=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 287603 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=(nil) Attempting backtrace. You
mysql 4.0.18 crashing on startup
I've got a customer with a mysql that is crashing on startup. MySQL is 4.0.18 with InnoDB tables on a Linux 2.4.26 system. MySQL binaries are Linux x86 glibc static gcc from mysql.com Can anyone divine what might have happened to cause this? thanks, baba resolve_stack_dump gives: 0x8071f44 handle_segfault + 420 0x82a0e38 pthread_sighandler + 184 0x818cfeb trx_undo_rec_get_partial_row + 443 0x81302af row_purge_parse_undo_rec + 863 0x813033f row_purge + 127 0x81304d7 row_purge_step + 23 0x8115aa3 que_run_threads + 755 0x8189520 trx_purge + 2304 0x811b8dd srv_master_thread + 1389 0x829e5ec pthread_start_thread + 220 0x82c7dea thread_start + 4 the .err log shows this, ever repeating: 050222 10:49:57 mysqld started 050222 10:49:57 InnoDB: Started /usr/sbin/mysqld: ready for connections. Version: '4.0.18-standard' socket: '/tmp/mysql4.sock' port: 3306 mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=1048576 read_buffer_size=4190208 max_used_connections=0 max_connections=35 threads_connected=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 287603 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=(nil) Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xbfebf5b8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x8071f44 0x82a0e38 0x818cfc1 0x81302af 0x813033f 0x81304d7 0x8115aa3 0x8189520 0x811bb6f 0x829e5ec 0x82c7dea New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 1 050222 10:49:58 mysqld restarted 050222 10:49:59 InnoDB: Started /usr/sbin/mysqld: ready for connections. Version: '4.0.18-standard' socket: '/tmp/mysql4.sock' port: 3306 mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=1048576 read_buffer_size=4190208 max_used_connections=0 max_connections=35 threads_connected=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 287603 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=(nil) Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xbfebf5b8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x8071f44 0x82a0e38 0x818cfc1 0x81302af 0x813033f 0x81304d7 0x8115aa3 0x8189520 0x811bb6f 0x829e5ec 0x82c7dea New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 1 050222 10:50:00 mysqld restarted 050222 10:50:01 InnoDB: Started /usr/sbin/mysqld: ready for connections. Version: '4.0.18-standard' socket: '/tmp/mysql4.sock' port: 3306 mysqld got signal 11; -- Baba Buehler - NetBotz, Inc. - [EMAIL PROTECTED] -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
corruption after database restore
I'm having a corruption problem after doing a backup and then a restore with ibbackup (v1.40). After restoring from the backup, when mysqld starts I get: 041006 07:46:53 mysqld started InnoDB: Warning: an inconsistent page in the doublewrite buffer InnoDB: space id 0 page number 5877102, 7'th page in dblwr buf. InnoDB: Warning: an inconsistent page in the doublewrite buffer InnoDB: space id 0 page number 5877103, 8'th page in dblwr buf. InnoDB: Warning: an inconsistent page in the doublewrite buffer InnoDB: space id 0 page number 5877104, 9'th page in dblwr buf. InnoDB: Warning: an inconsistent page in the doublewrite buffer InnoDB: space id 0 page number 5877105, 10'th page in dblwr buf. InnoDB: Warning: an inconsistent page in the doublewrite buffer InnoDB: space id 0 page number 5877106, 11'th page in dblwr buf. InnoDB: Error: tablespace size stored in header is 6029312 pages, but InnoDB: the sum of data file sizes is 5767168 pages 041006 7:46:54 InnoDB: Started /usr/sbin/mysqld: ready for connections. Version: '4.0.18-standard' socket: '/tmp/mysql4.sock' port: 3306 Then when queries start hitting the database, I start getting a lot of errors like (full backtrace included below): InnoDB: Assertion failure in thread 36874 in file fil0fil.c line 1204 I've restored from this backup multiple times with the same results, so I'm presuming its the backup itself that is corrupt. Does anyone have any ideas on what might cause ibbackup to corrupt files, as the backup appeared to complete successfully? Thanks, baba InnoDB: Error: trying to access page number 141623 in space 0 InnoDB: which is outside the tablespace bounds. InnoDB: Byte offset 0, len 16384, i/o type 10 041006 7:49:38 InnoDB: Assertion failure in thread 36874 in file fil0fil.c line 1204 InnoDB: Failing assertion: 0 InnoDB: We intentionally generate a memory trap. InnoDB: Send a detailed bug report to [EMAIL PROTECTED] mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=1048576 read_buffer_size=4190208 max_used_connections=1 max_connections=35 threads_connected=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 287603 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x87b2120 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xbfe7df28, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x8071f44 handle_segfault + 420 0x82a0e38 pthread_sighandler + 184 0x81edc77 fil_io + 1287 0x81b416b buf_read_page_low + 203 0x81b45c9 buf_read_page + 41 0x81a6fb8 buf_page_get_gen + 888 0x8167771 btr_cur_open_at_rnd_pos + 897 0x8171a4e btr_estimate_number_of_different_key_vals + 670 0x8101c99 dict_update_statistics_low + 89 0x8101d04 dict_update_statistics + 20 0x80f78f8 dict_table_get_and_increment_handle_count + 552 0x80ccc5b open__11ha_innobasePCciUi + 203 0x80c6b54 ha_open__7handlerPCcii + 36 0x8094595 openfrm__FPCcT0UiUiUiP8st_table + 5317 0x8090d27 open_unireg_entry__FP3THDP8st_tablePCcN22 + 87 0x8090178 open_table__FP3THDPCcN21Pb + 888 0x809102b open_tables__FP3THDP13st_table_list + 75 0x8091308 open_and_lock_tables__FP3THDP13st_table_list + 24 0x807ccb3 mysql_execute_command__Fv + 947 0x8080735 mysql_parse__FP3THDPcUi + 149 0x807be13 dispatch_command__F19enum_server_commandP3THDPcUi + 1443 0x807b85e do_command__FP3THD + 158 0x807b088 handle_one_connection + 648 0x829e5ec pthread_start_thread + 220 0x82c7dea thread_start + 4 New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd-query at 0x87b2af8 = SELECT DISTINCT serial FROM alert WHERE resolve_time IS NULL thd-thread_id=3 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. -- Baba Buehler - NetBotz, Inc. - [EMAIL PROTECTED] -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
