Hi John,Thank you! Can you give any insight as to what is the cause of the problem, or point me to any Basho Documentation detailing this?
Cheers, Bryan On 7/8/13 8:22 AM, John Caprice wrote:
Hey Bryan,This indicates a problem with the Bitcask data file. That data file, according to the second error report, was truncated. You more than likely did not experience any data loss as this would affect only a single replica, and only those contained in that data file. To be safe, you can repair the partition by attaching to Riak and running:riak_kv:repair(22835963083295358096932575511191922182123945984).after which you can detach from Riak with ctrl-d and monitor the status of the repair in riak-admin transfers. This command will read-repair any lost replicas due to the data file truncation.Thanks, John CapriceOn Mon, Jul 8, 2013 at 8:11 AM, Bryan Hughes <[email protected] <mailto:[email protected]>> wrote:Andrew, Thanks for the tip on how to use Google. :) But that was not my original question. I wanted to understand in more detail from the Basho folks what 2013-07-07 12:51:42 =ERROR REPORT==== Hintfile './data/bitcask/22835963083295358096932575511191922182123945984/3.bitcask.hint' contains pointer 16555635 566 that is greater than total data size 16556032 and 2013-07-07 12:54:43 =ERROR REPORT==== Bad datafile entry, discarding(383/566 bytes) meant to my system. For example, did I lose data and if so, how do I know what data was lost? More importantly is if this is data lost, how did it happen. I ran fsck on all the disks and checked the health of the system - which is all good. The later information was included for completeness. Bryan On 7/8/13 12:49 AM, Andrew Berman wrote:Bryan, What version of Erlang? You should check this out: https://github.com/basho/riak_kv/issues/411 BTW - Google is your friend, which is how I found the above issue :) --Andrew On Sun, Jul 7, 2013 at 3:01 PM, Bryan Hughes <[email protected] <mailto:[email protected]>> wrote: Hi Mark, DOH - sorry for the lack of detail. Didnt have enough coffee this morning. OS: CentOS release 6.3 (Final) Riak: Riak 1.2.1 Hadnt had a chance to upgrade to 1.3 yet. Got the node back up - but not entirely sure why which is a little concerning. Been verifying the data, and everything looks intact. When I try to run riak-admin status, I get the following (note I am not entirely sure this was the case when we first set the node up): $ riak-admin status Status failed, see log for details The logs shows: 2013-07-07 14:55:03.858 [error] <0.12982.0>@riak_kv_console:status:173 Status failed error:function_clause 2013-07-07 14:55:03.858 [error] emulator Error in process <0.12983.0> on node '[email protected] <mailto:[email protected]>' with exit value: {badarg,[{erlang,system_info,[global_heaps_size],[]},{riak_kv_stat,system_stats,0,[{file,"src/riak_kv_stat.erl"},{line,421}]},{riak_kv_stat,produce_stats,0,[{file,"src/riak_kv_stat.erl"},{line,320}]},{timer,tc,3,[{file,"timer... This is on a dev cluster with an out-of-the box configuration using bitcask. Thanks! Bryan On 7/7/13 2:51 PM, Mark Phillips wrote:Hi Bryan, I remember seeing something similar on the list a while ago. I'll dig through the archives (Riak.markmail.org <http://Riak.markmail.org>) if I have a few minutes later tonight. In the mean time, what version of Riak is this? And what OS? Mark On Sunday, July 7, 2013, Bryan Hughes wrote: Anyone familiar with this error message? 2013-07-07 12:51:42 =ERROR REPORT==== Hintfile './data/bitcask/22835963083295358096932575511191922182123945984/3.bitcask.hint' contains pointer 16555635 566 that is greater than total data size 16556032 2013-07-07 12:51:45 =ERROR REPORT==== Hintfile './data/bitcask/114179815416476790484662877555959610910619729920/3.bitcask.hint' contains pointer 17817310 567 <tel:17817310%20567> that is greater than total data size 17817600 2013-07-07 12:51:46 =ERROR REPORT==== Hintfile './data/bitcask/159851741583067506678528028578343455274867621888/3.bitcask.hint' contains pointer 7573448 567 <tel:7573448%20567> that is greater than total data size 7573504 2013-07-07 12:51:46 =ERROR REPORT==== Bad datafile entry 1: {ok,<<131,104,2,109,0,0,0,9,65,80,73,67,79,85,78,84,83,109,0,0,0,33,55,56,54,57,52,49,56,49,94,103,111,115,101,114,118,105,99,101,95,99>>} 2013-07-07 12:51:56 =ERROR REPORT==== Hintfile './data/bitcask/730750818665451459101842416358141509827966271488/3.bitcask.hint' contains pointer 13229833 581 that is greater than total data size 13230080 2013-07-07 12:52:05 =ERROR REPORT==== Hintfile './data/bitcask/1187470080331358621040493926581979953470445191168/3.bitcask.hint' contains pointer 23465420 578 that is greater than total data size 23465984 2013-07-07 12:52:06 =ERROR REPORT==== Hintfile './data/bitcask/1210306043414653979137426502093171875652569137152/3.bitcask.hint' contains pointer 27733824 578 that is greater than total data size 27734016 2013-07-07 12:52:07 =ERROR REPORT==== Hintfile './data/bitcask/1233142006497949337234359077604363797834693083136/3.bitcask.hint' contains pointer 15014008 578 <tel:15014008%20578> that is greater than total data size 15014586 2013-07-07 12:54:43 =ERROR REPORT==== Bad datafile entry, discarding(383/566 bytes) 2013-07-07 12:54:45 =ERROR REPORT==== Bad datafile entry, discarding(276/567 bytes) 2013-07-07 12:54:46 =ERROR REPORT==== Bad datafile entry, discarding(42/567 bytes) 2013-07-07 12:54:57 =ERROR REPORT==== Bad datafile entry, discarding(233/581 bytes) 2013-07-07 12:55:06 =ERROR REPORT==== Bad datafile entry, discarding(550/578 bytes) 2013-07-07 12:55:07 =ERROR REPORT==== Bad datafile entry, discarding(178/578 bytes) 2013-07-07 12:56:00 =ERROR REPORT==== Error in process <0.1536.0> on node '[email protected]' with exit value: {badarg,[{erlang,system_info,[global_heaps_size],[]},{riak_kv_stat,system_stats,0,[{file,"src/riak_kv_stat.erl"},{line,421}]},{riak_kv_stat,produce_stats,0,[{file,"src/riak_kv_stat.erl"},{line,320}]},{timer,tc,3,[{file,"timer...--Bryan Hughes *Go Factory* http://www.go-factory.net /"Internet Class, Enterprise Grade"/--Bryan Hughes CTO and Founder / *Go Factory* (415) 515-7916 <tel:%28415%29%20515-7916> http://www.go-factory.net /"Internet Class, Enterprise Grade"/ _______________________________________________ riak-users mailing list [email protected] <mailto:[email protected]> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com_______________________________________________ riak-users mailing list [email protected] <mailto:[email protected]> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
