Hi Doug, i added ThriftBroker.API.Logging=true to hypertable.cfg file and cap push_config. and then restart the ThriftBrokers. ThriftBroker.log file as following,
hadoop59 's ThriftBroker log: tcmalloc: large alloc 1668055040 bytes == 0x34ce000 @ tcmalloc: large alloc 1668055040 bytes == 0x34ce000 @ 1386592292.534763000 API namespace_open: API namespace_open: namespace name=crawler id=3625439156 latency=0 1386592892 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:385): Received error: is_create=1 - Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=172.16.200.153:38060 - HYPERTABLE request timeout 1386592892 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:402): Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from= 172.16.200.153:38060 - HYPERTABLE request timeout 1386592892 ERROR ThriftBroker : hql_exec (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:584): virtual void Hypertable::ThriftBroker::ServerHandler::hql_exec(Hypertable::ThriftGen::HqlResult&, Hypertable::ThriftGen::Namespace, const Hypertable::String&, bool, bool) namespace=3625439156 hql=select guid FROM urls where row=' http://washingtonpost.com/business/capitalbusiness/kastle-systems-has-seen-video-mobile-technology-transform-its-security-business/2013/12/07/62a071c4-5de8-11e3-95c2-13623eb2b0e1_story.html?tid=hpModule_1728cf4a-8a79-11e2-98d9-3012c1cd8d1e' cell_limit 1 noflush=0 unbuffered=0: Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=172.16.200.153:38060 - HYPERTABLE request timeout at bool Hypertable::TableScanner::next(Hypertable::Cell&) (/root/src/hypertable/src/cc/Hypertable/Lib/TableScanner.cc:82) 1386592892.639819000 API namespace_close: API namespace_close: namespace=3625439156 latency=0 1386592892.640398000 API namespace_open: API namespace_open: namespace name=crawler id=2342619381 latency=0 1386592902.650809000 API namespace_open: API namespace_open: namespace name=crawler id=970823060 latency=0 hadoop152 ThriftBroker log: 1386592891.786474000 API namespace_close: API namespace_close: namespace=2282353234 latency=0 1386592891 ERROR ThriftBroker : TSocket::peek() recv() <Host: 172.16.200.30 Port: 56469>Connection reset by peer 1386592891 ERROR ThriftBroker : TThreadedServer client died: recv(): Connection reset by peer 1386592901 ERROR ThriftBroker : flush (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:183): Hypertable::Exception: - DFS BROKER i/o error at void Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsync*) (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:210) Failed: (2013-12-09#ali_ata_bak,parametre,-9223372036854775806) - DFS BROKER i/o error 1386592901 ERROR ThriftBroker : hql_exec (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:584): virtual void Hypertable::ThriftBroker::ServerHandler::hql_exec(Hypertable::ThriftGen::HqlResult&, Hypertable::ThriftGen::Namespace, const Hypertable::String&, bool, bool) namespace=1660590572 hql=insert into sorgukayit values ('2013-12-09#ali_ata_bak','parametre','172.16.59.67') noflush=0 unbuffered=0: Hypertable::Exception: - DFS BROKER i/o error at void Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsync*) (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:210) 1386593491 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:596) Received response for non-pending event (id=19,version=1,total_len=91) and now we can not insert or select on ht shell. 2013/12/6 Doug Judd <[email protected]> > Hi Hatice, > > Add the following property to your hypertable.cfg file: > > ThriftBroker.API.Logging=true > > and then restart the ThriftBrokers and wait for the problem to happen > again. Once it does, send me the ThriftBroker.log file of the ThriftBroker > that crashed. It will include more information that will help me diagnose > the problem. Thanks! > > - Doug > > > On Fri, Dec 6, 2013 at 6:09 AM, Hatice Çobanoğlu < > [email protected]> wrote: > >> Hi everyone, >> i have 7 thriftboker in cluster. and i've problem oen of them as >> following. >> >> 1386334385 ERROR ThriftBroker : get_namespace >> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad >> namespace id - 0 >> 1386334385 ERROR ThriftBroker : get_namespace >> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad >> namespace id - 0 >> 1386334385 ERROR ThriftBroker : get_namespace >> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad >> namespace id - 0 >> 1386334385 ERROR ThriftBroker : get_namespace >> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad >> namespace id - 0 >> 1386334385 ERROR ThriftBroker : get_namespace >> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad >> namespace id - 0 >> 1386334385 ERROR ThriftBroker : get_namespace >> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad >> namespace id - 0 >> 1386334385 ERROR ThriftBroker : get_namespace >> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad >> namespace id - 0 >> 1386335515 ERROR ThriftBroker : TThreadedServer exception: St9bad_alloc: >> std::bad_alloc >> 1386335532 ERROR ThriftBroker : TThreadedServer exception: St9bad_alloc: >> std::bad_alloc >> 1386335593 ERROR ThriftBroker : TThreadedServer exception: St9bad_alloc: >> std::bad_alloc >> 1386335653 ERROR ThriftBroker : TThreadedServer exception: St9bad_alloc: >> std::bad_alloc >> 1386336456 INFO ThriftBroker : >> (/root/src/hypertable/src/cc/Common/Config.cc:591) Initializing >> ThriftBroker (Hypertable 0.9.7.13 (v0.9$ >> CPU cores count=24 >> >> when have these problems, restart thriftbroker which closed and system >> continue. why thrift give this error log? >> not: connect to thrift with 32 client. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Hypertable Development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/hypertable-dev. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Doug Judd > CEO, Hypertable Inc. > > -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/hypertable-dev. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/hypertable-dev. For more options, visit https://groups.google.com/groups/opt_out.
