Hi there, I just had my mysql dB crash really badly on a small server I'm running and wonder if someone could point me in the right direction. I'm pretty basic with mysql... I set up master/slave replication to another site, so this could be a backup, so long as the errors were not replicated over there.
Mysql boots with these types of errors recorded in syslog. Also is the my.cnf If you know of ideas that could be of help, then I woul dbe really grateful. I've been trying to get this to run for 4 hours solid. Best regards, Simon My.cnf ( My server only has 512Mb of RAM so mysql has to use not a lot). # grep -v ^$ /etc/mysql/my.cnf|grep -v ^# [client] port = 3306 socket = /var/run/mysqld/mysqld.sock [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp skip-external-locking bind-address = 127.0.0.1 max_allowed_packet = 1M key_buffer = 16K sort_buffer_size = 64K read_buffer_size = 256K thread_stack = 64K thread_cache_size = 8 myisam-recover = BACKUP max_connections = 34 log_slow_queries = /var/log/mysql/mysql-slow.log query_cache_limit = 64K query_cache_size = 2M expire_logs_days = 10 max_binlog_size = 100M table_open_cache = 4 table_cache = 48 read_rnd_buffer_size = 256K net_buffer_length = 128K server-id = 1 log-bin=/var/log/mysql/mysql-bin.log innodb_flush_log_at_trx_commit = 2 max_binlog_size = 15M expire_logs_days = 2 [mysqldump] quick quote-names max_allowed_packet = 1M [mysql] [isamchk] key_buffer = 16M !includedir /etc/mysql/conf.d/ [myisamchk] key_buffer_size = 386K sort_buffer_size = 64K read_buffer_size = 256K Syslog records 36000 lines of errors from mysql and below please find a snip it: Aug 28 11:50:14 emailserver mysqld_safe: Number of processes running now: 0 Aug 28 11:50:14 emailserver mysqld_safe: mysqld restarted Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Warning] option 'thread_stack': unsigned value 65536 adjusted to 131072 Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead. Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Note] Plugin 'FEDERATED' is disabled. Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: The InnoDB memory heap is disabled Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Mutexes and rw_locks use GCC atomic builtins Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Compressed tables use zlib 1.2.3.4 Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Using Linux native AIO Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Initializing buffer pool, size = 128.0M Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Completed initialization of buffer pool Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: highest supported file format is Barracuda. Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Waiting for the background threads to start Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 InnoDB: 5.5.31 started; log sequence number 1211163767 Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Recovering after a crash using /var/log/mysql/mysql-bin Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Starting crash recovery... Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Crash recovery finished. Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306 Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] - '127.0.0.1' resolves to '127.0.0.1'; Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Server socket created on IP: '127.0.0.1'. Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Event Scheduler: Loaded 0 events Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] /usr/sbin/mysqld: ready for connections. Aug 28 11:50:15 emailserver mysqld: Version: '5.5.31-1~dotdeb.0-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian) Aug 28 11:50:15 emailserver mysqld: InnoDB: Dump of the tablespace extent descriptor: len 40; hex 00000000000000000000000001de00000000016600000002aaaaaaaaaaaaaaaaeaaaeafaabaabbaa; asc f ; Aug 28 11:50:15 emailserver mysqld: InnoDB: Serious error! InnoDB is trying to free page 815 Aug 28 11:50:15 emailserver mysqld: InnoDB: though it is already marked as free in the tablespace! Aug 28 11:50:15 emailserver mysqld: InnoDB: The tablespace free space info is corrupt. Aug 28 11:50:15 emailserver mysqld: InnoDB: You may need to dump your InnoDB tables and recreate the whole Aug 28 11:50:15 emailserver mysqld: InnoDB: database! Aug 28 11:50:15 emailserver mysqld: InnoDB: Please refer to Aug 28 11:50:15 emailserver mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html Aug 28 11:50:15 emailserver mysqld: InnoDB: about forcing recovery. Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 InnoDB: Assertion failure in thread 139642230933248 in file fsp0fsp.c line 3309 Aug 28 11:50:15 emailserver mysqld: InnoDB: We intentionally generate a memory trap. Aug 28 11:50:15 emailserver mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com. Aug 28 11:50:15 emailserver mysqld: InnoDB: If you get repeated assertion failures or crashes, even Aug 28 11:50:15 emailserver mysqld: InnoDB: immediately after the mysqld startup, there may be Aug 28 11:50:15 emailserver mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to Aug 28 11:50:15 emailserver mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html Aug 28 11:50:15 emailserver mysqld: InnoDB: about forcing recovery. Aug 28 11:50:15 emailserver mysqld: 09:50:15 UTC - mysqld got signal 6 ; Aug 28 11:50:15 emailserver mysqld: This could be because you hit a bug. It is also possible that this binary Aug 28 11:50:15 emailserver mysqld: or one of the libraries it was linked against is corrupt, improperly built, Aug 28 11:50:15 emailserver mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Aug 28 11:50:15 emailserver mysqld: We will try our best to scrape up some info that will hopefully help Aug 28 11:50:15 emailserver mysqld: diagnose the problem, but since we have already crashed, Aug 28 11:50:15 emailserver mysqld: something is definitely wrong and this may fail. Aug 28 11:50:15 emailserver mysqld: Aug 28 11:50:15 emailserver mysqld: key_buffer_size=16384 Aug 28 11:50:15 emailserver mysqld: read_buffer_size=262144 Aug 28 11:50:15 emailserver mysqld: max_used_connections=0 Aug 28 11:50:15 emailserver mysqld: max_threads=34 Aug 28 11:50:15 emailserver mysqld: thread_count=0 Aug 28 11:50:15 emailserver mysqld: connection_count=0 Aug 28 11:50:15 emailserver mysqld: It is possible that mysqld could use up to Aug 28 11:50:15 emailserver mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11283 K bytes of memory Aug 28 11:50:15 emailserver mysqld: Hope that's ok; if not, decrease some variables in the equation. Aug 28 11:50:15 emailserver mysqld: Aug 28 11:50:15 emailserver mysqld: Thread pointer: 0x0 Aug 28 11:50:15 emailserver mysqld: Attempting backtrace. You can use the following information to find out Aug 28 11:50:15 emailserver mysqld: where mysqld died. If you see no messages after this, something went Aug 28 11:50:15 emailserver mysqld: terribly wrong... Aug 28 11:50:15 emailserver mysqld: stack_bottom = 0 thread_stack 0x20000 Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f011094b14e] Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x493)[0x7f0110802843] Aug 28 11:50:15 emailserver mysqld: /lib/libpthread.so.0(+0xeff0)[0x7f010ffbbff0] Aug 28 11:50:15 emailserver mysqld: /lib/libc.so.6(gsignal+0x35)[0x7f010e86d1b5] Aug 28 11:50:15 emailserver mysqld: /lib/libc.so.6(abort+0x180)[0x7f010e86ffc0] Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(+0x682405)[0x7f0110a6b405] Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(+0x682735)[0x7f0110a6b735] Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(+0x6394b9)[0x7f0110a224b9] Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(+0x6d1b0f)[0x7f0110abab0f] Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(+0x6d2438)[0x7f0110abb438] Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(+0x6c8255)[0x7f0110ab1255] Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(+0x613435)[0x7f01109fc435] Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(+0x60656c)[0x7f01109ef56c] Aug 28 11:50:15 emailserver mysqld: /usr/sbin/mysqld(+0x6092a3)[0x7f01109f22a3] Aug 28 11:50:15 emailserver mysqld: /lib/libpthread.so.0(+0x68ca)[0x7f010ffb38ca] Aug 28 11:50:15 emailserver mysqld: /lib/libc.so.6(clone+0x6d)[0x7f010e90ab6d] Aug 28 11:50:15 emailserver mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Aug 28 11:50:15 emailserver mysqld: information that should help you find out what is causing the crash. Aug 28 11:50:15 emailserver mysqld_safe: Number of processes running now: 0 Aug 28 11:50:15 emailserver mysqld_safe: mysqld restarted Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Warning] option 'thread_stack': unsigned value 65536 adjusted to 131072 Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead. Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Note] Plugin 'FEDERATED' is disabled. Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: The InnoDB memory heap is disabled Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Mutexes and rw_locks use GCC atomic builtins Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Compressed tables use zlib 1.2.3.4 Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Using Linux native AIO Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Initializing buffer pool, size = 128.0M Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Completed initialization of buffer pool Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: highest supported file format is Barracuda. Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Waiting for the background threads to start Aug 28 11:50:16 emailserver mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 InnoDB: 5.5.31 started; log sequence number 1211163767 Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Recovering after a crash using /var/log/mysql/mysql-bin Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Starting crash recovery... Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Crash recovery finished. Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306 Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] - '127.0.0.1' resolves to '127.0.0.1'; Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Server socket created on IP: '127.0.0.1'. Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Event Scheduler: Loaded 0 events Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] /usr/sbin/mysqld: ready for connections. Aug 28 11:50:17 emailserver mysqld: Version: '5.5.31-1~dotdeb.0-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian) Aug 28 11:50:17 emailserver mysqld: InnoDB: Dump of the tablespace extent descriptor: len 40; hex 00000000000000000000000001de00000000016600000002aaaaaaaaaaaaaaaaeaaaeafaabaabbaa; asc f ; Aug 28 11:50:17 emailserver mysqld: InnoDB: Serious error! InnoDB is trying to free page 815 Aug 28 11:50:17 emailserver mysqld: InnoDB: though it is already marked as free in the tablespace! Aug 28 11:50:17 emailserver mysqld: InnoDB: The tablespace free space info is corrupt. Aug 28 11:50:17 emailserver mysqld: InnoDB: You may need to dump your InnoDB tables and recreate the whole Aug 28 11:50:17 emailserver mysqld: InnoDB: database! Aug 28 11:50:17 emailserver mysqld: InnoDB: Please refer to Aug 28 11:50:17 emailserver mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html Aug 28 11:50:17 emailserver mysqld: InnoDB: about forcing recovery. Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 InnoDB: Assertion failure in thread 140076906338048 in file fsp0fsp.c line 3309 Aug 28 11:50:17 emailserver mysqld: InnoDB: We intentionally generate a memory trap. Aug 28 11:50:17 emailserver mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com. Aug 28 11:50:17 emailserver mysqld: InnoDB: If you get repeated assertion failures or crashes, even Aug 28 11:50:17 emailserver mysqld: InnoDB: immediately after the mysqld startup, there may be Aug 28 11:50:17 emailserver mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to Aug 28 11:50:17 emailserver mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html Aug 28 11:50:17 emailserver mysqld: InnoDB: about forcing recovery. Aug 28 11:50:17 emailserver mysqld: 09:50:17 UTC - mysqld got signal 6 ; Aug 28 11:50:17 emailserver mysqld: This could be because you hit a bug. It is also possible that this binary Aug 28 11:50:17 emailserver mysqld: or one of the libraries it was linked against is corrupt, improperly built, Aug 28 11:50:17 emailserver mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Aug 28 11:50:17 emailserver mysqld: We will try our best to scrape up some info that will hopefully help Aug 28 11:50:17 emailserver mysqld: diagnose the problem, but since we have already crashed, Aug 28 11:50:17 emailserver mysqld: something is definitely wrong and this may fail. Aug 28 11:50:17 emailserver mysqld: Aug 28 11:50:17 emailserver mysqld: key_buffer_size=16384 Aug 28 11:50:17 emailserver mysqld: read_buffer_size=262144 Aug 28 11:50:17 emailserver mysqld: max_used_connections=0 Aug 28 11:50:17 emailserver mysqld: max_threads=34 Aug 28 11:50:17 emailserver mysqld: thread_count=0 Aug 28 11:50:17 emailserver mysqld: connection_count=0 Aug 28 11:50:17 emailserver mysqld: It is possible that mysqld could use up to Aug 28 11:50:17 emailserver mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11283 K bytes of memory Aug 28 11:50:17 emailserver mysqld: Hope that's ok; if not, decrease some variables in the equation. Aug 28 11:50:17 emailserver mysqld: Aug 28 11:50:17 emailserver mysqld: Thread pointer: 0x0 Aug 28 11:50:17 emailserver mysqld: Attempting backtrace. You can use the following information to find out Aug 28 11:50:17 emailserver mysqld: where mysqld died. If you see no messages after this, something went Aug 28 11:50:17 emailserver mysqld: terribly wrong... Aug 28 11:50:17 emailserver mysqld: stack_bottom = 0 thread_stack 0x20000 Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f664541014e] Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x493)[0x7f66452c7843] Aug 28 11:50:17 emailserver mysqld: /lib/libpthread.so.0(+0xeff0)[0x7f6644a80ff0] Aug 28 11:50:17 emailserver mysqld: /lib/libc.so.6(gsignal+0x35)[0x7f66433321b5] Aug 28 11:50:17 emailserver mysqld: /lib/libc.so.6(abort+0x180)[0x7f6643334fc0] Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(+0x682405)[0x7f6645530405] Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(+0x682735)[0x7f6645530735] Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(+0x6394b9)[0x7f66454e74b9] Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(+0x6d1b0f)[0x7f664557fb0f] Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(+0x6d2438)[0x7f6645580438] Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(+0x6c8255)[0x7f6645576255] Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(+0x613435)[0x7f66454c1435] Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(+0x60656c)[0x7f66454b456c] Aug 28 11:50:17 emailserver mysqld: /usr/sbin/mysqld(+0x6092a3)[0x7f66454b72a3] Aug 28 11:50:17 emailserver mysqld: /lib/libpthread.so.0(+0x68ca)[0x7f6644a788ca] Aug 28 11:50:17 emailserver mysqld: /lib/libc.so.6(clone+0x6d)[0x7f66433cfb6d] Aug 28 11:50:17 emailserver mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Aug 28 11:50:17 emailserver mysqld: information that should help you find out what is causing the crash.
