Hi list readers, I've checked out the latest source code and re-ran my indexing application for the wikipedia corpus. It seems the code is much more stable now and the database can withstand insertions for much longer periods of time. But the usual bug is still there:
1237586373 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:350) Starting Major Compaction of wikipedia_index[..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].](default) 1237586377 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 298054887 bytes 1237586377 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2076) Cleaning log (threshold=3114270720) 1237586397 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 298089509 bytes 1237586397 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2076) Cleaning log (threshold=200000000) 1237586417 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 298089509 bytes 1237586437 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 298089509 bytes 1237586457 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 298089509 bytes 1237586477 WARN Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/ScannerMap.cc:82) Destroying scanner 3 because it has not been used in 120000 milliseconds 1237586477 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 298089509 bytes 1237586497 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 298089509 bytes 1237586517 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 298089509 bytes 1237586537 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 298089509 bytes 1237586557 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 298089509 bytes 1237586558 ERROR Hypertable.RangeServer : handle_exceptions (/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:47): Hypertable::Exception: waiting for previous buffer - HYPERTABLE request timeout at void Hypertable::TableMutator::wait_for_previous_buffer(Hypertable::Timer&) (/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:300) at bool Hypertable::TableMutatorCompletionCounter::wait_for_completion(Hypertable::Timer&) (/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:71): 1237586558 ERROR Hypertable.RangeServer : update_files_column (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:114): Problem updating 'Files' column of METADATA: Hypertable::Exception: waiting for previous buffer - HYPERTABLE request timeout at void Hypertable::TableMutator::wait_for_previous_buffer(Hypertable::Timer&) (/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:300) at bool Hypertable::TableMutatorCompletionCounter::wait_for_completion(Hypertable::Timer&) (/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:71): When restarting RangeServer after this crash the following messages appear in the log 1237587141 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1722) Successfully replay loaded range wikipedia_index[..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].] 1237587141 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:230) Loading CellStore /hypertable/tables/wikipedia_index/default/AB2A0D28DE6B77FFDD6C72AF/cs1 1237587141 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1722) Successfully replay loaded range wikipedia_index[http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700]...mythology] 1237587141 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:230) Loading CellStore /hypertable/tables/wikipedia_index/default/AB2A0D28DE6B77FFDD6C72AF/cs1 1237587141 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1722) Successfully replay loaded range wikipedia_index[mythology..��] 1237587142 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:520) Replayed 40 blocks of updates from '/hypertable/servers/172.16.0.19_38060/log/user' 1237587142 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:978) Replayed 12403 updates (12 blocks) from split log '/hypertable/servers/172.16.0.19_38060/log/82DF41632DA476065CE29979' into wikipedia_index[..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].] 1237587142 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:834) Restored range state to SPLIT_LOG_INSTALLED (split point='community' split log='/hypertable/servers/172.16.0.19_38060/log/82DF41632DA476065CE29979') 1237587145 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:350) Starting Major Compaction of wikipedia_index[..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].](default) 1237587149 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:515) Finished Compaction of wikipedia_index[..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].](default) 1237587149 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:726) Reporting newly split off range wikipedia_index[community..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].] to Master 1237587149 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:798) Loading range: {TableIdentifier: name='wikipedia_index' id=4 generation=1} {RangeSpec: start='community' end='http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].'} 1237587149 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:230) Loading CellStore /hypertable/tables/wikipedia_index/default/A4CDB32FC10A364A233092C2/cs1 1237587149 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:978) Replayed 12403 updates (12 blocks) from split log '/hypertable/servers/172.16.0.19_38060/log/82DF41632DA476065CE29979' into wikipedia_index[community..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].] 1237587149 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:958) Successfully loaded range wikipedia_index[community..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].] 1237587149 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:381) Split Complete. New Range end_row= 1237587165 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019) Memory Usage: 15055844 bytes 1237587165 INFO Hypertable.RangeServer : (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2076) Cleaning log (threshold=200000000) It seems to me that the major compaction of the range started but never finished in the first run. I'm attaching logs to this message as they may be helpful. The symptoms are as before, CPU usage drops to zero, memory usage is at 25% and everything timeouts after a while. Anything I can do to help debugging this error, please bug me ? Mateusz --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
logs.tgz
Description: GNU Zip compressed data
