Hey Guys
Yesterday I sent an email, about '1 client is using or hasn't closed the table
properly', but the problem is actually bigger than I realised.
We run Mysql replication, and on the second node, Mysql is crashing with
'mysqld got signal 6' every so often.
Other than the drives, we have replaced the hardware, and still the problem
persists.
Googling, I thought it was a memory exhaustion issue, so I started playing with
the following variables and values.
key_buffer_size=8384512
max_connections=41
But it still crashes
Mar 27 12:36:53 xyz-web02 mysqld: key_buffer_size=8384512
Mar 27 12:36:53 xyz-web02 mysqld: read_buffer_size=131072
Mar 27 12:36:53 xyz-web02 mysqld: max_used_connections=3
Mar 27 12:36:53 xyz-web02 mysqld: max_threads=41
Mar 27 12:36:53 xyz-web02 mysqld: thread_count=1
Mar 27 12:36:53 xyz-web02 mysqld: connection_count=1
Mar 27 12:36:53 xyz-web02 mysqld: It is possible that mysqld could use up to
Mar 27 12:36:53 xyz-web02 mysqld: key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 97828 K bytes of memory
xyz-web02:# free -m
total used free shared buffers cached
Mem: 12038 11966 71 0 117 11378
-/+ buffers/cache: 469 11568
Swap: 2047 0 2047
We running Debian Squeeze 64bit.
I have attached the full crash message.
If anyone can help, I would be *most* grateful (If you are in South Africa, I
will buy you a meal, let alone a beer :)
Kindest Regards
Brent Clark
*** glibc detected *** /usr/sbin/mysqld: double free or corruption (!prev):
0x00007f5e38003b60 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71bd6)[0x7f5e4575dbd6]
/lib/libc.so.6(cfree+0x6c)[0x7f5e4576294c]
/usr/sbin/mysqld(_ZN15Query_log_eventD0Ev+0x20)[0x7f5e474f2ed0]
/usr/sbin/mysqld(handle_slave_sql+0x835)[0x7f5e47567255]
/lib/libpthread.so.0(+0x68ca)[0x7f5e46c738ca]
/lib/libc.so.6(clone+0x6d)[0x7f5e457bb86d]
======= Memory map: ========
7f5e30000000-7f5e30021000 rw-p 00000000 00:00 0
7f5e30021000-7f5e34000000 ---p 00000000 00:00 0
7f5e38000000-7f5e38158000 rw-p 00000000 00:00 0
7f5e38158000-7f5e3c000000 ---p 00000000 00:00 0
7f5e3e019000-7f5e3e01a000 ---p 00000000 00:00 0
7f5e3e01a000-7f5e3e81a000 rw-p 00000000 00:00 0
7f5e3e81a000-7f5e3e81b000 ---p 00000000 00:00 0
7f5e3e81b000-7f5e3f01b000 rw-p 00000000 00:00 0
7f5e3f01b000-7f5e3f01c000 ---p 00000000 00:00 0
7f5e3f01c000-7f5e3f81c000 rw-p 00000000 00:00 0
7f5e3f81c000-7f5e3f81d000 ---p 00000000 00:00 0
7f5e3f81d000-7f5e4001d000 rw-p 00000000 00:00 0
7f5e4001d000-7f5e4001e000 ---p 00000000 00:00 0
7f5e4001e000-7f5e40a24000 rw-p 00000000 00:00 0
7f5e40c47000-7f5e40c48000 ---p 00000000 00:00 0
7f5e40c48000-7f5e41448000 rw-p 00000000 00:00 0
7f5e41448000-7f5e41449000 ---p 00000000 00:00 0
7f5e41449000-7f5e41c49000 rw-p 00000000 00:00 0
7f5e41c49000-7f5e41c4a000 ---p 00000000 00:00 0
7f5e41c4a000-7f5e448c4000 rw-p 00000000 00:00 0
7f5e448c4000-7f5e448d0000 r-xp 00000000 fd:00 328894
/lib/libnss_files-2.11.3.so
7f5e448d0000-7f5e44acf000 ---p 0000c000 fd:00 328894
/lib/libnss_files-2.11.3.so
7f5e44acf000-7f5e44ad0000 r--p 0000b000 fd:00 328894
/lib/libnss_files-2.11.3.so
7f5e44ad0000-7f5e44ad1000 rw-p 0000c000 fd:00 328894
/lib/libnss_files-2.11.3.so
7f5e44ad1000-7f5e44adb000 r-xp 00000000 fd:00 328896
/lib/libnss_nis-2.11.3.so
7f5e44adb000-7f5e44cda000 ---p 0000a000 fd:00 328896
/lib/libnss_nis-2.11.3.so
7f5e44cda000-7f5e44cdb000 r--p 00009000 fd:00 328896
/lib/libnss_nis-2.11.3.so
7f5e44cdb000-7f5e44cdc000 rw-p 0000a000 fd:00 328896
/lib/libnss_nis-2.11.3.so
7f5e44cdc000-7f5e44ce3000 r-xp 00000000 fd:00 328892
/lib/libnss_compat-2.11.3.so
7f5e44ce3000-7f5e44ee2000 ---p 00007000 fd:00 328892
/lib/libnss_compat-2.11.3.so
7f5e44ee2000-7f5e44ee3000 r--p 00006000 fd:00 328892
/lib/libnss_compat-2.11.3.so
7f5e44ee3000-7f5e44ee4000 rw-p 00007000 fd:00 328892
/lib/libnss_compat-2.11.3.so
7f5e44eeb000-7f5e44eec000 ---p 00000000 00:00 0
7f5e44eec000-7f5e456ec000 rw-p 00000000 00:00 0
7f5e456ec000-7f5e45845000 r-xp 00000000 fd:00 328847
/lib/libc-2.11.3.so
7f5e45845000-7f5e45a44000 ---p 00159000 fd:00 328847
/lib/libc-2.11.3.so
7f5e45a44000-7f5e45a48000 r--p 00158000 fd:00 328847
/lib/libc-2.11.3.so
7f5e45a48000-7f5e45a49000 rw-p 0015c000 fd:00 328847
/lib/libc-2.11.3.so
7f5e45a49000-7f5e45a4e000 rw-p 00000000 00:00 0
7f5e45a4e000-7f5e45a64000 r-xp 00000000 fd:00 328801
/lib/libgcc_s.so.1
7f5e45a64000-7f5e45c63000 ---p 00016000 fd:00 328801
/lib/libgcc_s.so.1
7f5e45c63000-7f5e45c64000 rw-p 00015000 fd:00 328801
/lib/libgcc_s.so.1
7f5e45c64000-7f5e45ce4000 r-xp 00000000 fd:00 328889
/lib/libm-2.11.3.so
7f5e45ce4000-7f5e45ee4000 ---p 00080000 fd:00 328889
/lib/libm-2.11.3.so
7f5e45ee4000-7f5e45ee5000 r--p 00080000 fd:00 328889
/lib/libm-2.11.3.so
7f5e45ee5000-7f5e45ee6000 rw-p 00081000 fd:00 328889
/lib/libm-2.11.3.so
7f5e45ee6000-7f5e45fdc000 r-xp 00000000 fd:02 173092
/usr/lib/libstdc++.so.6.0.13
7f5e45fdc000-7f5e461dc000 ---p 000f6000 fd:02 173092
/usr/lib/libstdc++.so.6.0.13
7f5e461dc000-7f5e461e3000 r--p 000f6000 fd:02 173092
/usr/lib/libstdc++.so.6.0.13
7f5e461e3000-7f5e461e5000 rw-p 000fd000 fd:02 173092
/usr/lib/libstdc++.so.6.0.13
7f5e461e5000-7f5e461fa000 rw-p 00000000 00:00 0
7f5e461fa000-7f5e4620f000 r-xp 00000000 fd:00 328891
/lib/libnsl-2.11.3.so
7f5e4620f000-7f5e4640e000 ---p 00015000 fd:00 328891
/lib/libnsl-2.11.3.so
7f5e4640e000-7f5e4640f000 r--p 00014000 fd:00 328891
/lib/libnsl-2.11.3.so
7f5e4640f000-7f5e46410000 rw-p 00015000 fd:00 328891
/lib/libnsl-2.11.3.so
7f5e46410000-7f5e46412000 rw-p 00000000 00:00 0
7f5e46412000-7f5e4641a000 r-xp 00000000 fd:00 328885
/lib/libcrypt-2.11.3.so
7f5e4641a000-7f5e46619000 ---p 00008000 fd:00 328885
/lib/libcrypt-2.11.3.so
7f5e46619000-7f5e4661a000 r--p 00007000 fd:00 328885
/lib/libcrypt-2.11.3.so
7f5e4661a000-7f5e4661b000 rw-p 00008000 fd:00 328885
/lib/libcrypt-2.11.3.so
7f5e4661b000-7f5e46649000 rw-p 00000000 00:00 0
7f5e46649000-7f5e4664b000 r-xp 00000000 fd:00 328887
/lib/libdl-2.11.3.so
7f5e4664b000-7f5e4684b000 ---p 00002000 fd:00 328887
/lib/libdl-2.11.3.so
7f5e4684b000-7f5e4684c000 r--p 00002000 fd:00 328887
/lib/libdl-2.11.3.so
7f5e4684c000-7f5e4684d000 rw-p 00003000 fd:00 328887
/lib/libdl-2.11.3.so
7f5e4684d000-7f5e46855000 r-xp 00000000 fd:00 328825
/lib/libwrap.so.0.7.6
7f5e46855000-7f5e46a54000 ---p 00008000 fd:00 328825
/lib/libwrap.so.0.7.6
7f5e46a54000-7f5e46a56000 rw-p 00007000 fd:00 328825
/lib/libwrap.so.0.7.6
7f5e46a56000-7f5e46a6d000 r-xp 00000000 fd:02 173161
/usr/lib/libz.so.1.2.3.4
7f5e46a6d000-7f5e46c6c000 ---p 00017000 fd:02 173161
/usr/lib/libz.so.1.2.3.4
7f5e46c6c000-7f5e46c6d000 rw-p 00016000 fd:02 173161
/usr/lib/libz.so.1.2.3.4
7f5e46c6d000-7f5e46c84000 r-xp 00000000 fd:00 328787
/lib/libpthread-2.11.3.so
7f5e46c84000-7f5e46e83000 ---p 00017000 fd:00 328787
/lib/libpthread-2.11.3.so
7f5e46e83000-7f5e46e84000 r--p 00016000 fd:00 328787
/lib/libpthread-2.11.3.so
7f5e46e84000-7f5e46e85000 rw-p 00017000 fd:00 328787
/lib/libpthread-2.11.3.so
Mar 27 12:30:39 xyz-web02 mysqld: 10:30:39 UTC - mysqld got signal 6 ;
Mar 27 12:30:39 xyz-web02 mysqld: This could be because you hit a bug. It is
also possible that this binary
Mar 27 12:30:39 xyz-web02 mysqld: or one of the libraries it was linked against
is corrupt, improperly built,
Mar 27 12:30:39 xyz-web02 mysqld: or misconfigured. This error can also be
caused by malfunctioning hardware.
Mar 27 12:30:39 xyz-web02 mysqld: We will try our best to scrape up some info
that will hopefully help
Mar 27 12:30:39 xyz-web02 mysqld: diagnose the problem, but since we have
already crashed,
Mar 27 12:30:39 xyz-web02 mysqld: something is definitely wrong and this may
fail.
Mar 27 12:30:39 xyz-web02 mysqld:
Mar 27 12:30:39 xyz-web02 mysqld: key_buffer_size=8384512
Mar 27 12:30:39 xyz-web02 mysqld: read_buffer_size=131072
Mar 27 12:30:39 xyz-web02 mysqld: max_used_connections=2
Mar 27 12:30:39 xyz-web02 mysqld: max_threads=41
Mar 27 12:30:39 xyz-web02 mysqld: thread_count=1
Mar 27 12:30:39 xyz-web02 mysqld: connection_count=1
Mar 27 12:30:39 xyz-web02 mysqld: It is possible that mysqld could use up to
Mar 27 12:30:39 xyz-web02 mysqld: key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 97828 K bytes of memory
Mar 27 12:30:39 xyz-web02 mysqld: Hope that's ok; if not, decrease some
variables in the equation.
Mar 27 12:30:39 xyz-web02 mysqld:
Mar 27 12:30:39 xyz-web02 mysqld: Thread pointer: 0x7f5e38000970
Mar 27 12:30:39 xyz-web02 mysqld: Attempting backtrace. You can use the
following information to find out
Mar 27 12:30:39 xyz-web02 mysqld: where mysqld died. If you see no messages
after this, something went
Mar 27 12:30:39 xyz-web02 mysqld: terribly wrong...
Mar 27 12:30:39 xyz-web02 mysqld: stack_bottom = 7f5e47033848 thread_stack
0x30000
Mar 27 12:30:39 xyz-web02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x29)
[0x7f5e47714b59]
Mar 27 12:30:39 xyz-web02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x483)
[0x7f5e475294a3]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libpthread.so.0(+0xeff0) [0x7f5e46c7bff0]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(gsignal+0x35) [0x7f5e4571e1b5]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(abort+0x180) [0x7f5e45720fc0]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(+0x6837b) [0x7f5e4575437b]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(+0x71bd6) [0x7f5e4575dbd6]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(cfree+0x6c) [0x7f5e4576294c]
Mar 27 12:30:39 xyz-web02 mysqld:
/usr/sbin/mysqld(Query_log_event::~Query_log_event()+0x20) [0x7f5e474f2ed0]
Mar 27 12:30:39 xyz-web02 mysqld: /usr/sbin/mysqld(handle_slave_sql+0x835)
[0x7f5e47567255]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libpthread.so.0(+0x68ca) [0x7f5e46c738ca]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(clone+0x6d) [0x7f5e457bb86d]
Mar 27 12:30:39 xyz-web02 mysqld:
Mar 27 12:30:39 xyz-web02 mysqld: Trying to get some variables.
Mar 27 12:30:39 xyz-web02 mysqld: Some pointers may be invalid and cause the
dump to abort.
Mar 27 12:30:39 xyz-web02 mysqld: Query (0): is an invalid pointer
Mar 27 12:30:39 xyz-web02 mysqld: Connection ID (thread ID): 2
Mar 27 12:30:39 xyz-web02 mysqld: Status: NOT_KILLED
Mar 27 12:30:39 xyz-web02 mysqld:
Mar 27 12:30:39 xyz-web02 mysqld: The manual page at
http://dev.mysql.com/doc/mysql/en/crashing.html contains
Mar 27 12:30:39 xyz-web02 mysqld: information that should help you find out
what is causing the crash.
Mar 27 12:30:39 xyz-web02 mysqld_safe: Number of processes running now: 0
Mar 27 12:30:39 xyz-web02 mysqld_safe: mysqld restarted
Mar 27 12:30:39 xyz-web02 mysqld: 120327 12:30:39 [Warning] 'for replication
startup options' is deprecated and will be removed in a future release. Please
use ''CHANGE MASTER'' instead.
Mar 27 12:30:39 xyz-web02 mysqld: 120327 12:30:39 [Note] Plugin 'FEDERATED' is
disabled.
Mar 27 12:30:39 xyz-web02 mysqld: 120327 12:30:39 InnoDB: Initializing buffer
pool, size = 8.0M
Mar 27 12:30:39 xyz-web02 mysqld: 120327 12:30:39 InnoDB: Completed
initialization of buffer pool
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: Log scan progressed past the
checkpoint lsn 21 2468511949
Mar 27 12:30:39 xyz-web02 mysqld: 120327 12:30:39 InnoDB: Database was not
shut down normally!
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: Starting crash recovery.
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: Reading tablespace information from
the .ibd files...
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: Restoring possible half-written data
pages from the doublewrite
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: buffer...
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: Doing recovery: scanned up to log
sequence number 21 2468634155
Mar 27 12:30:40 xyz-web02 mysqld: 120327 12:30:40 InnoDB: Starting an apply
batch of log records to the database...
Mar 27 12:30:41 xyz-web02 mysqld: InnoDB: Progress in percents: 0 1 2 3 4 5 6 7
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
87 88 89 90 91 92 93 94 95 96 97 98 99
Mar 27 12:30:41 xyz-web02 mysqld: InnoDB: Apply batch completed
Mar 27 12:30:41 xyz-web02 mysqld: InnoDB: Last MySQL binlog file position 0
106, file name /var/log/mysql/mysql-bin.002274
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 InnoDB: Started; log
sequence number 21 2468634155
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 [Note] Recovering after a
crash using /var/log/mysql/mysql-bin
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 [Note] Starting crash
recovery...
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 [Note] Crash recovery
finished.
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 [Note] Slave SQL thread
initialized, starting replication in log 'mysql-bin.000845' at position
244888243, relay log '/var/lib/mysql/slave-relay.000262' position: 4602980
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 [Note] Event Scheduler:
Loaded 0 events
--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql
