Michael, I guess the table is corrupt and its row contains a wrong trx id. InnoDB tries to retrieve an earlier version of the row, but the roll ptr in the row is garbage too, and it contains a nonexistent rollback segment id.
" InnoDB: Error: trying to return an SQL NULL field in a non-null innoDB: column! Table name secret/tablename " The above is also a clear indication of table corruption. The NULL bit in an InnoDB column is set, though the field should never be NULL. You should upgrade to MySQL-4.0.17, which has better data page checksum checking. Do you get corruption with the same application on earlier Linux kernels, or could there be something wrong with Linux-2.4.23? Best regards, Heikki Innobase Oy http://www.innodb.com InnoDB - transactions, row level locking, and foreign keys for MySQL InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up MyISAM tables Order MySQL support from http://www.mysql.com/support/index.html /********************************************************************** Copies an undo record to heap. This function can be called if we know that the undo log record exists. */ trx_undo_rec_t* trx_undo_get_undo_rec_low( /*======================*/ /* out, own: copy of the record */ dulint roll_ptr, /* in: roll pointer to record */ mem_heap_t* heap) /* in: memory heap where copied */ { trx_undo_rec_t* undo_rec; ulint rseg_id; ulint page_no; ulint offset; page_t* undo_page; trx_rseg_t* rseg; ibool is_insert; mtr_t mtr; trx_undo_decode_roll_ptr(roll_ptr, &is_insert, &rseg_id, &page_no, &offset); rseg = trx_rseg_get_on_id(rseg_id); ... /********************************************************************** Looks for a rollback segment, based on the rollback segment id. */ trx_rseg_t* trx_rseg_get_on_id( /*===============*/ /* out: rollback segment */ ulint id) /* in: rollback segment id */ { trx_rseg_t* rseg; rseg = UT_LIST_GET_FIRST(trx_sys->rseg_list); ut_ad(rseg); while (rseg->id != id) { rseg = UT_LIST_GET_NEXT(rseg_list, rseg); ut_ad(rseg); } return(rseg); } (gdb) disassemble trx_rseg_get_on_id Dump of assembler code for function trx_rseg_get_on_id: 0x81fa930 <trx_rseg_get_on_id>: push %ebp 0x81fa931 <trx_rseg_get_on_id+1>: mov 0x83f3190,%eax 0x81fa936 <trx_rseg_get_on_id+6>: mov 0x24(%eax),%eax 0x81fa939 <trx_rseg_get_on_id+9>: mov %esp,%ebp 0x81fa93b <trx_rseg_get_on_id+11>: mov 0x8(%ebp),%edx 0x81fa93e <trx_rseg_get_on_id+14>: cmp %edx,(%eax) 0x81fa940 <trx_rseg_get_on_id+16>: je 0x81fa94c <trx_rseg_get_on_id+28> 0x81fa942 <trx_rseg_get_on_id+18>: mov 0xa0(%eax),%eax 0x81fa948 <trx_rseg_get_on_id+24>: cmp %edx,(%eax) 0x81fa94a <trx_rseg_get_on_id+26>: jne 0x81fa942 <trx_rseg_get_on_id+18> 0x81fa94c <trx_rseg_get_on_id+28>: mov %ebp,%esp 0x81fa94e <trx_rseg_get_on_id+30>: pop %ebp 0x81fa94f <trx_rseg_get_on_id+31>: ret End of assembler dump. (gdb) ................ List:MySQL General Discussion« Previous MessageNext Message » From:Michael BacarellaDate:January 26 2004 6:21pm Subject:Re: MySQL 3.23.58 seg faults occasionally > > the crashes below happen in independent areas of code. The 2 first are > > inside InnoDB, and the third inside MySQL. This looks like random thread > > crashes, or random memory corruption. > > My colleague tells me that the third stack trace (the one inside MySQL) > is from a RESET MASTER. This server used to be a slave that was promoted > to master. My apologies for not making this distinction. > > The ones inside InnoDB have been triggered by querying disk-resident > low use tables. We can get SIGSEGV almost every hour (from an hourly maintenance script). This is the stack trace: 0x80c23d5 handle_segfault__Fi + 425 0x40022f54 _end + 935506260 0x823d4a8 trx_rseg_get_on_id + 24 0x823952d trx_undo_get_undo_rec_low + 45 0x823977d trx_undo_get_undo_rec + 49 0x82399c0 trx_undo_prev_version_build + 548 0x81f3f35 row_vers_build_for_consistent_read + 641 0x81e9d5e row_sel_build_prev_vers_for_mysql + 226 0x81ecda4 row_search_for_mysql + 6580 0x8111152 general_fetch__11ha_innobasePcUiUi + 322 0x8111373 rnd_next__11ha_innobasePc + 83 0x8103da6 rr_sequential__FP14st_read_record + 150 0x80e74d9 sub_select__FP4JOINP13st_join_tableb + 341 0x80e7190 do_select__FP4JOINPt4List1Z4ItemP8st_tableP9Procedure + 412 0x80df9d7 mysql_select__FP3THDP13st_table_listRt4List1Z4ItemP4ItemP8st_orderT4T3T4UiP1 3select_result + 4167 0x80c8aba mysql_execute_command__Fv + 806 0x80cbd88 mysql_parse__FP3THDPcUi + 72 0x80c7c74 do_command__FP3THD + 1324 Attempting to convert the table that caused the crash to MyISAM resulted in: InnoDB: Error: trying to return an SQL NULL field in a non-null innoDB: column! Table name secret/tablename Also, duplicate keys. We're deleting the table and recreating it. Any thoughts? > > You are running a relatively new Linux kernel, 2.4.23. Did the crashes start > > when you upgraded Linux? > > Well, maybe. This is a new machine. The previous machine was Red Hat 8.0's > default kernel, which is 2.4.18 + RH patches. We were also running 3.23.56 > which didn't seg fault like this. > > >From the log, MySQL says: > > > It is possible that mysqld could use up to > > key_buffer_size + (record_buffer + sort_buffer)*max_connections = 1884024 K > > bytes of memory > > I never remember if the above calculation is usable or not. > > basedir=/usr/local/mysql/ > user=mysql > server-id=3 > log-bin=/dblog/dbms3-bin > > log-slow-queries = /usr/local/mysql/data/dbms3.slow_queries > set-variable = long_query_time=5 > > innodb_data_home_dir = > innodb_data_file_path = /dev/md0:76319Mraw > > set-variable = innodb_buffer_pool_size=800M > set-variable = innodb_additional_mem_pool_size=50M > > innodb_log_group_home_dir = /dblog > innodb_log_arch_dir = /dblog > innodb_log_archive=0 > > set-variable = innodb_log_files_in_group=3 > set-variable = innodb_log_file_size=1.5G > set-variable = innodb_log_buffer_size=8M > > innodb_flush_log_at_trx_commit=0 > set-variable = innodb_file_io_threads=4 > set-variable = innodb_lock_wait_timeout=50 > > set-variable = wait_timeout=360 > set-variable = tmp_table_size=8000000 > set-variable = max_connections=415 > set-variable = table_cache=256 > set-variable = sort_buffer=4M > set-variable = key_buffer=128M > set-variable = thread_cache_size=415 > set-variable = max_connect_errors=50000 > set-variable = thread_stack=256k -- Michael Bacarella 24/7 phone: 1-646-641-8662 Netgraft Corporation http://netgraft.com/ -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
