I'll build you a debug version which will give better debugging output. If you could upgrade to that version, and try again, that will help me figure out what's going on. I'll try to get the to you later today.
- Doug On Thu, May 16, 2013 at 12:01 AM, Hatice Çobanoğlu < [email protected]> wrote: > Hi Doug, > > following output. > gdb /opt/hypertable/current/bin/Hypertable.RangeServer ~/core > > GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04 > Copyright (C) 2012 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later < > http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-linux-gnu". > For bug reporting instructions, please see: > <http://bugs.launchpad.net/gdb-linaro/>... > Reading symbols from /opt/hypertable/ > 0.9.7.5/bin/Hypertable.RangeServer...done. > > warning: core file may not match specified executable file. > [New LWP 3183] > [New LWP 3186] > [New LWP 3188] > [New LWP 3189] > [New LWP 3187] > [New LWP 3191] > [New LWP 3184] > [New LWP 3195] > [New LWP 3196] > [New LWP 3202] > [New LWP 3185] > [New LWP 3200] > [New LWP 3190] > [New LWP 3193] > [New LWP 3198] > [New LWP 3062] > [New LWP 3192] > [New LWP 3064] > [New LWP 3203] > [New LWP 3208] > [New LWP 3154] > [New LWP 3211] > [New LWP 3057] > [New LWP 3204] > [New LWP 3206] > [New LWP 3199] > [New LWP 3151] > [New LWP 3173] > [New LWP 3052] > [New LWP 3210] > [New LWP 3081] > [New LWP 3159] > [New LWP 3080] > [New LWP 3165] > [New LWP 3205] > [New LWP 3137] > [New LWP 2994] > [New LWP 3103] > [New LWP 3213] > [New LWP 3068] > [New LWP 3060] > [New LWP 3147] > [New LWP 3073] > [New LWP 3146] > [New LWP 3055] > [New LWP 3156] > [New LWP 3071] > [New LWP 3079] > [New LWP 3157] > [New LWP 3086] > [New LWP 3145] > [New LWP 3090] > [New LWP 3141] > [New LWP 3215] > [New LWP 3220] > [New LWP 3070] > [New LWP 3104] > [New LWP 3131] > [New LWP 3172] > [New LWP 3061] > [New LWP 3152] > [New LWP 3102] > [New LWP 3113] > [New LWP 3083] > [New LWP 3170] > [New LWP 3118] > [New LWP 3114] > [New LWP 3093] > [New LWP 3066] > [New LWP 3051] > [New LWP 3150] > [New LWP 3067] > [New LWP 3120] > [New LWP 3116] > [New LWP 3182] > [New LWP 3084] > [New LWP 3108] > [New LWP 3207] > [New LWP 3139] > [New LWP 3096] > [New LWP 3155] > [New LWP 3143] > [New LWP 3212] > [New LWP 3091] > [New LWP 3097] > [New LWP 3162] > [New LWP 3178] > [New LWP 3130] > [New LWP 3197] > [New LWP 3125] > [New LWP 3088] > [New LWP 3140] > [New LWP 3176] > [New LWP 3209] > [New LWP 3133] > [New LWP 3177] > [New LWP 3171] > [New LWP 3148] > [New LWP 3174] > [New LWP 3214] > [New LWP 3087] > [New LWP 3217] > [New LWP 3126] > [New LWP 3107] > [New LWP 3164] > [New LWP 3136] > [New LWP 3053] > [New LWP 3168] > [New LWP 3180] > [New LWP 3132] > [New LWP 3122] > [New LWP 3095] > [New LWP 3094] > [New LWP 3092] > [New LWP 3166] > [New LWP 3221] > [New LWP 3169] > [New LWP 3160] > [New LWP 3158] > [New LWP 3163] > [New LWP 3216] > [New LWP 3082] > [New LWP 3219] > [New LWP 3109] > [New LWP 3089] > [New LWP 3063] > [New LWP 3105] > [New LWP 3106] > [New LWP 3111] > [New LWP 3218] > [New LWP 3181] > [New LWP 3134] > [New LWP 3179] > [New LWP 3194] > [New LWP 3138] > [New LWP 3112] > [New LWP 3117] > [New LWP 3167] > [New LWP 3056] > [New LWP 3129] > [New LWP 3175] > [New LWP 3069] > [New LWP 3072] > [New LWP 3153] > [New LWP 3050] > [New LWP 3121] > [New LWP 3085] > [New LWP 3115] > [New LWP 3074] > [New LWP 3119] > [New LWP 3058] > [New LWP 3149] > [New LWP 3161] > [New LWP 3110] > [New LWP 3127] > [New LWP 3144] > [New LWP 3059] > [New LWP 3075] > [New LWP 3065] > [New LWP 3201] > [New LWP 3078] > [New LWP 3123] > [New LWP 3128] > [New LWP 3142] > [New LWP 3054] > [New LWP 3135] > [New LWP 3124] > Core was generated by `/opt/hypertable/current/bin/Hypertable.Master > --pidfile /opt/hypertable/current'. > Program terminated with signal 11, Segmentation fault. > #0 0x0000000000583604 in > boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::gregorian::bad_day_of_month> > >::rethrow() const () > > > (gdb) thread apply all where > > Thread 167 (LWP 3124): > #0 0x00007ff1dd37dd84 in ?? () > #1 0x000047d100000000 in ?? () > #2 0x00000000023cf060 in ?? () > #3 0x00000000023cf038 in ?? () > #4 0x00000000000ffb10 in ?? () > #5 0x0000000002564478 in ?? () > #6 0x00000000005a6f8a in > Hypertable::TableInfo::add_range(boost::intrusive_ptr<Hypertable::Range>&, > bool) () > #7 0x0000000000000000 in ?? () > > Thread 166 (LWP 3135): > #0 0x00007ff1dd37dd84 in ?? () > #1 0x000047d100000000 in ?? () > #2 0x00000000023cf060 in ?? () > #3 0x00000000023cf038 in ?? () > #4 0x00000000000ffb10 in ?? () > #5 0x0000000003ccbb98 in ?? () > #6 0x00000000005a6f8a in > Hypertable::TableInfo::add_range(boost::intrusive_ptr<Hypertable::Range>&, > bool) () > #7 0x0000000000000000 in ?? () > > Thread 165 (LWP 3054): > ---Type <return> to continue, or q <return> to quit--- > > > > > 2013/5/15 Doug Judd <[email protected]> > >> Did the RangeServer that crashed create a core file? If so, can you load >> the core file into gdb as follows: >> >> gdb /opt/hypertable/current/bin/Hypertable.RangeServer <core-file> >> >> and then run the following command and post the output: >> >> (gdb) thread apply all where >> >> - Doug >> >> >> On Wed, May 15, 2013 at 4:48 AM, Hatice Çobanoğlu < >> [email protected]> wrote: >> >>> sorry Doug, we can't. our servers working on only local. Do you have >>> another resolve solition this problem ? what can i do? >>> the worst-case scenario is "cap cleandb" for our. >>> >>> >>> >>> >>> >>> 2013/5/14 Hatice Çobanoğlu <[email protected]> >>> >>>> Hi Doug >>>> To access our servers or database hasn't been possible before. But i >>>> will ask my projects manager tomorrow. >>>> >>>> Bye >>>> Hatice >>>> 14 Mayıs 2013 Salı tarihinde Doug Judd adlı kullanıcı şöyle yazdı: >>>> >>>> Hi Hatice, >>>>> >>>>> It looks like one of the RangeServers asserted. The best way for me >>>>> to get to the bottom of this would be for me to log in and work on your >>>>> database directly. Would that be possible? >>>>> >>>>> - Doug >>>>> >>>>> >>>>> >>>>> On Tue, May 14, 2013 at 12:00 AM, Hatice Çobanoğlu < >>>>> [email protected]> wrote: >>>>> >>>>> hi Doug, >>>>> yes, file 26 is largest numerically name file in the mml directory. i >>>>> copy file 27 into /hypertable/servers/master/log/mml and then start >>>>> hypertable. I waited for a while. the system still has problems >>>>> recovering. i send you logs file in attach. >>>>> >>>>> (Did i copy file 27 into >>>>> /opt/hypertable/current/run/log_backup/mml/master_38050/ ? ) >>>>> >>>>> >>>>> 2013/5/14 Doug Judd <[email protected]> >>>>> >>>>> Hi Hatice, >>>>> >>>>> Attached is MML file 27 (SHA1 >>>>> bdbad7794a9d40e6f7def224562cb05d19ad2e06). Before you install it, make >>>>> sure that the largest numerically name file in the mml directory >>>>> (/hypertable/servers/master/log/mml) is 26. If it is not 26, then stop >>>>> Hypertable and send me the largest numerically named fragment file in that >>>>> directory and *do not* restart Hypertable. Each time you start >>>>> Hypertable it will create a new file which I'll have to modify and send >>>>> back to you. If 26 is the largest file, then copy the attached file into >>>>> /hypertable/servers/master/log/mml and then start Hypertable. Be prepared >>>>> to send me the log files if the system still has problems recovering. >>>>> >>>>> - Doug >>>>> >>>>> >>>>> >>>>> On Mon, May 13, 2013 at 7:08 AM, Hatice Çobanoğlu < >>>>> [email protected]> wrote: >>>>> >>>>> i send you MML file in attach. >>>>> >>>>> thanks >>>>> Hatice >>>>> >>>>> >>>>> 2013/5/13 Doug Judd <[email protected]> >>>>> >>>>> Ok, I'm not sure why that command didn't work for you, but I managed >>>>> to run it successfully. I'll send you the fixed MML file today. BTW, can >>>>> you also send me the contents of this directory: >>>>> >>>>> /hypertable/servers/master/log/mml >>>>> >>>>> Also, don't try to restart Hypertable until I get you the patched MML >>>>> file. >>>>> >>>>> - Doug >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, May 13, 2013 at 6:36 AM, Hatice Çobanoğlu < >>>>> [email protected]> wrote: >>>>> >>>>> yes, we'r sure. i send you log files in attach. >>>>> >>>>> thanks, >>>>> Hatice >>>>> >>>>> >>>>> 2013/5/13 Doug Judd <[email protected]> >>>>> >>>>> Are you sure all of the Hypertable processes are stopped? Be sure to >>>>> run 'cap stop' and then verify that all the Hypertable processes are >>>>> stopped with: >>>>> >>>>> cap shell >>>>> cap> ps auxww | fgrep -i hyper | fgrep -v java >>>>> >>>>> If not, manually kill the processes. If you're still getting the same >>>>> thing, send me the output of: >>>>> >>>>> hadoop fs >>>>> >>>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Hypertable Development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/hypertable-dev?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> >> >> -- >> Doug Judd >> CEO, Hypertable Inc. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Hypertable Development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/hypertable-dev?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/hypertable-dev?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- Doug Judd CEO, Hypertable Inc. -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/hypertable-dev?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
