On Dec 14, 2008, at 7:19 PM, Doug Judd wrote:
> There is an assert statement in the code that is firing, causing the > segfault. We need to figure out why we can't create the DFS file / > hypertable/tables/dtn_daily2/default/D4356DC81572CB43970A2BB4/ > cs132. Are there any ERRORS in the DfsBroker log? Also, are there > any errors in your DFS logs (e.g. Hadoop or KFS) ? > That looks (at least from the log file perspective) so much like the bug I was having. Will recheck if it still exists on my end as well later today. Mateusz > - Doug > > On Sun, Dec 14, 2008 at 10:15 AM, EricHolmberg <[email protected] > > wrote: > > > Are there any ERROR messages in the Hypertable.RangeServer.log > file ? > > Yes, here's a condensed version of the log. Essentially, everything > is going fine for several hours and then I get an "Error creating > cellstore: Error creating DFS file ..." > message followed by a "Hypertable::Exception: Problem compacting > locality group to file '/hypertable/tables/dtn_daily2/default/ > D4356DC81572CB43970A2BB4/cs132' - But that's unpossible!" message. > After 1507 error messages, the rangeserver segfaults. > > I have trimmed the import file down from 12 GB to 500KB and it imports > fine. I haven't done any analysis to see where the breaking point is. > > Hypertable.RangerServer.log > ----------------------------------------- > [Import going well up to this point having run for several hours -- > memory usage 6 GB out of 12 GB physical] > > 1229052129 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/RangeServer.cc: > 1828) Memory Usage: 5525900676 bytes > 1229052129 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/RangeServer.cc: > 1885) Cleaning log (threshold=20000000000) > 1229052129 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 330) Starting Minor Compaction of dtn_daily2 > [WVLMD_2008-01-19_2007-01-08..XNZDP_2007-04-21_2007-01-22](default) > 1229052129 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 464) Finished Compaction of dtn_daily2 > [WVLMD_2008-01-19_2007-01-08..XNZDP_2007-04-21_2007-01-22](default) > 1229052133 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 330) Starting Minor Compaction of dtn_daily2 > [..AUAQY_2007-05-19_2007-02-26](default) > 1229052134 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 330) Starting Minor Compaction of dtn_daily2 > [AUAQY_2007-05-19_2007-02-26..BENPD_2007-04-21_2007-01-22](default) > 1229052134 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 464) Finished Compaction of dtn_daily2[..AUAQY_2007-05-19_2007-02-26] > (default) > 1229052135 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 464) Finished Compaction of dtn_daily2 > [AUAQY_2007-05-19_2007-02-26..BENPD_2007-04-21_2007-01-22](default) > 1229052136 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 321) Starting Merging Compaction of dtn_daily2 > [BROCE_2007-03-17_2007-02-27..COHBZ_2007-02-17_2007-02-15](default) > 1229052137 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 330) Starting Minor Compaction of dtn_daily2 > [COHBZ_2007-02-17_2007-02-15..DGXMZ_2007-01-20_2007-01-18](default) > 1229052137 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 464) Finished Compaction of dtn_daily2 > [COHBZ_2007-02-17_2007-02-15..DGXMZ_2007-01-20_2007-01-18](default) > 1229052140 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 330) Starting Minor Compaction of dtn_daily2 > [EKGF_2007-07-21_2007-01-26..EVTH_2007-08-18_2007-03-09](default) > 1229052140 ERROR Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/CellStoreV0.cc: > 143) Error creating cellstore: Error creating DFS file: /hypertable/ > tables/dtn_daily2/default/D4356DC81572CB43970A2BB4/cs132 > 1229052140 ERROR Hypertable.RangeServer : run_compaction (/tmp/ > hypertable/hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/ > AccessGroup.cc:469): Hypertable::Exception: Problem compacting > locality group to file '/hypertable/tables/dtn_daily2/default/ > D4356DC81572CB43970A2BB4/cs132' - But that's unpossible! > at void Hypertable::AccessGroup::run_compaction(bool) (/tmp/ > hypertable/hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/ > AccessGroup.cc:361) > 1229052141 INFO Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/AccessGroup.cc: > 330) Starting Minor Compaction of dtn_daily2 > [EVTH_2007-08-18_2007-03-09..FLKPR_2007-04-21_2007-01-01](default) > 1229052141 ERROR Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/CellStoreV0.cc: > 143) Error creating cellstore: Error creating DFS file: /hypertable/ > tables/dtn_daily2/default/AAFF5D1E6FEFC5F318AD9553/cs132 > 1229052141 ERROR Hypertable.RangeServer : run_compaction (/tmp/ > hypertable/hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/ > AccessGroup.cc:469): Hypertable::Exception: Problem compacting > locality group to file '/hypertable/tables/dtn_daily2/default/ > AAFF5D1E6FEFC5F318AD9553/cs132' - But that's unpossible! > > [1500 similar lines to the previous values] > > 1229052374 ERROR Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/CellStoreV0.cc: > 143) Error creating cellstore: Error creating DFS file: /hypertable/ > tables/dtn_daily2/default/307F679AFE61680363E259E1/cs177 > 1229052374 ERROR Hypertable.RangeServer : run_compaction (/tmp/ > hypertable/hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/ > AccessGroup.cc:469): Hypertable::Exception: Problem compacting > locality group to file '/hypertable/tables/dtn_daily2/default/ > 307F679AFE61680363E259E1/cs177' - But that's unpossible! > at void Hypertable::AccessGroup::run_compaction(bool) (/tmp/ > hypertable/hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/ > AccessGroup.cc:361) > 1229052375 ERROR Hypertable.RangeServer : (/tmp/hypertable/ > hypertable-0.9.1.0-alpha/src/cc/Hypertable/RangeServer/Range.cc:401) > Problem creating log directory '/hypertable/servers/ > 192.168.1.50_38060/ > log/263A42B6B05C6F318F2B2DFF': Error removing DFS directory: / > hypertable/servers/192.168.1.50_38060/log/263A42B6B05C6F318F2B2DFF > > KABOOM! > > syslog > ----------------------------------------- > Hypertable.Rang[16923]: segfault at 0 ip 0000000000607912 sp > 000000004dcbea30 error 6 in Hypertable.RangeServer[400000+41f000] > > Configuration values (all values not shown are default) > ----------------------------------------------------------------------------- > Hypertable.RangeServer.AccessGroup.MaxMemory=20000000 > Hypertable.RangeServer.MaintenanceThreads=1 > > System > -------------------- > Ubuntu 8.10, amd64, 2.6.27-9-generic kernel > Boost 1.35 > Running using the local DFS > > Thanks, > > Eric > > > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
