Is there anything in the Hypertable.RangeServer.log file for 111.1111.111.111 that would indicate why it disconnected? If you post all of your log files we can take a look.
- Doug On Tue, Jul 31, 2012 at 7:03 AM, Kenny F. <[email protected]> wrote: > RangeServer crashes often: in 2-4 hours > > actions after crash: > >hypertable/0.9.6.0/bin/ht stop-servers > Killing ThriftBroker.pid 17175 > */opt/hypertable/0.9.6.0/bin/ht-env.sh: line 67: kill: (17175) - No such > process * > Shutdown master complete > Sending shutdown command > *Unable to establish connection to range server * > ... > sometimes: *Waiting for range server to shutdown... > Waiting for range server to shutdown... > Waiting for range server to shutdown... > Waiting for range server to shutdown... > Waiting for range server to shutdown... > Waiting for range server to shutdown...* > > > when I try to restart severs: > >/hypertable/0.9.6.0/bin/ht start all-servers local > ... > Started Hypertable.RangeServer > *Waiting for ThriftBroker to come up... > Waiting for ThriftBroker to come up... > Waiting for ThriftBroker to come up... > Waiting for ThriftBroker to come up... > Waiting for ThriftBroker to come up... > Waiting for ThriftBroker to come up... > Waiting for ThriftBroker to come up... > Waiting for ThriftBroker to come up... > ERROR: ThriftBroker did not come up* > > > Master Logs: > 1343746823 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343746823 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343746824 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:218) > Dropping OperationCollectGarbage because another one is outstanding > 1343746824 INFO Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) > Entering GatherStatistics-2372 state=INITIAL > 1343746824 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343746824 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > sh: dot: not found > 1343746824 ERROR Hypertable.Master : > (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) > rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", > "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile > 1343746824 INFO Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) > Leaving GatherStatistics-2372 > 1343746824 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343746824 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343746825 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343746825 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343746826 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343746826 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343746827 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343746827 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > > ThriftBroker Logs: > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343743796 INFO ThriftBroker : > (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: > type=DISCONNECT "COMM connect error" from=111.1111.111.111:38111; Problem > connecting to Root RangeServer, > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - > COMM not connected > 1343743796 WARN ThriftBroker : > (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) > Comm::send_request to rs1 failed - COMM not connected > .... > 1343747503 ERROR ThriftBroker : TThreadedServer client died: write() > send(): Broken pipe > 1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: > ::ffff:127.0.0.1 Port: 34830>Broken pipe > 1343747503 ERROR ThriftBroker : TThreadedServer client died: write() > send(): Broken pipe > 1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: > ::ffff:127.0.0.1 Port: 34835>Broken pipe > 1343747503 ERROR ThriftBroker : TThreadedServer client died: write() > send(): Broken pipe > 1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: > ::ffff:127.0.0.1 Port: 34712>Broken pipe > 1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: > ::ffff:127.0.0.1 Port: 34720>Broken pipe > 1343747503 ERROR ThriftBroker : TThreadedServer client died: write() > send(): Broken pipe > 1343747503 ERROR ThriftBroker : TThreadedServer client died: write() > send(): Broken pipe > 1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: > ::ffff:127.0.0.1 Port: 34750>Broken pipe > 1343747503 ERROR ThriftBroker : TThreadedServer client died: write() > send(): Broken pipe > 1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: > ::ffff:127.0.0.1 Port: 34653>Broken pipe > 1343747503 ERROR ThriftBroker : TThreadedServer client died: write() > send(): Broken pipe > 1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: > ::ffff:127.0.0.1 Port: 34788>Broken pipe > 1343747503 ERROR ThriftBroker : TThreadedServer client died: write() > send(): Broken pipe > 1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: > ::ffff:127.0.0.1 Port: 34808>Broken pipe > ... > > -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/hypertable-dev/-/bN3Xud3yvcoJ. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > -- Doug Judd CEO, Hypertable Inc. -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
