RangeServer crashes often: in 2-4 hours
actions after crash:
>hypertable/0.9.6.0/bin/ht stop-servers
Killing ThriftBroker.pid 17175
*/opt/hypertable/0.9.6.0/bin/ht-env.sh: line 67: kill: (17175) - No such
process *
Shutdown master complete
Sending shutdown command
*Unable to establish connection to range server *
...
sometimes: *Waiting for range server to shutdown...
Waiting for range server to shutdown...
Waiting for range server to shutdown...
Waiting for range server to shutdown...
Waiting for range server to shutdown...
Waiting for range server to shutdown...*
when I try to restart severs:
>/hypertable/0.9.6.0/bin/ht start all-servers local
...
Started Hypertable.RangeServer
*Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
ERROR: ThriftBroker did not come up*
Master Logs:
1343746823 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343746823 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343746824 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:218)
Dropping OperationCollectGarbage because another one is outstanding
1343746824 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57)
Entering GatherStatistics-2372 state=INITIAL
1343746824 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343746824 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
sh: dot: not found
1343746824 ERROR Hypertable.Master :
(/root/src/hypertable/src/cc/Common/FileUtils.cc:451)
rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg",
"/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
1343746824 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100)
Leaving GatherStatistics-2372
1343746824 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343746824 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343746825 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343746825 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343746826 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343746826 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343746827 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343746827 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
ThriftBroker Logs:
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343743796 INFO ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event:
type=DISCONNECT "COMM connect error" from=111.1111.111.111:38111; Problem
connecting to Root RangeServer,
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1343743796 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
....
1343747503 ERROR ThriftBroker : TThreadedServer client died: write()
send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host:
::ffff:127.0.0.1 Port: 34830>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write()
send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host:
::ffff:127.0.0.1 Port: 34835>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write()
send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host:
::ffff:127.0.0.1 Port: 34712>Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host:
::ffff:127.0.0.1 Port: 34720>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write()
send(): Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write()
send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host:
::ffff:127.0.0.1 Port: 34750>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write()
send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host:
::ffff:127.0.0.1 Port: 34653>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write()
send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host:
::ffff:127.0.0.1 Port: 34788>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write()
send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host:
::ffff:127.0.0.1 Port: 34808>Broken pipe
...
--
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/hypertable-dev/-/bN3Xud3yvcoJ.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en.
