Looks like the database was shut down uncleanly. It appears that the meta log is corrupt due to the HDFS fsync issue. You can fix the problem with the following sequence of commands:
cap stop cap cleandb cap start That should clear the database and bring it back up in a working state. - Doug On Tue, Dec 15, 2009 at 10:21 PM, Nilesh K <[email protected]> wrote: > Upon starting from shell the output is like this > +++++++++++++++++++++++++++++++++++++++++++++++ > [r...@localhost conf]# cap shell > * executing `shell' > ==================================================================== > Welcome to the interactive Capistrano shell! This is an experimental > feature, and is liable to change in future releases. Type 'help' for > a summary of how to use the shell. > -------------------------------------------------------------------- > cap> start > [establishing connection(s) to 192.168.2.69, 192.168.2.85, > 192.168.2.86] > *** [err :: 192.168.2.86] start: missing job name > *** [err :: 192.168.2.86] Try `start --help' for more information. > *** [err :: 192.168.2.85] start: missing job name > *** [err :: 192.168.2.85] Try `start --help' for more information. > *** [err :: 192.168.2.69] start: missing job name > *** [err :: 192.168.2.69] Try `start --help' for more information. > error: failed: "sh -c 'start'" on > 192.168.2.69,192.168.2.85,192.168.2.86 > > +++++++++++++++++++++++++++++++++++++++++++++++ > > and RangeServer log is like this > +++++++++++++++++++++++++++++++++++++++++++++++++++++ > 1260941869 NOTICE Hypertable.RangeServer : (/root/src/ > hypertable-0.9.2.7-alpha/src/cc/Common/Config.cc:409) Initializing > Hypertable.RangeServer (Hypertable 0.9.2.7 (tarball))... > CPU cores count=2 > CephBroker.MonAddr=10.0.1.245:6789 > CephBroker.Port=38030 > CephBroker.Workers=20 > DfsBroker.Host=localhost > DfsBroker.Local.Port=38030 > DfsBroker.Local.Root=fs/local > DfsBroker.Port=38030 > HdfsBroker.Port=38030 > HdfsBroker.Workers=20 > HdfsBroker.fs.default.name=hdfs://192.168.2.69:9000 > Hyperspace.GracePeriod=200000 > Hyperspace.KeepAlive.Interval=30000 > Hyperspace.Lease.Interval=1000000 > Hyperspace.Master.Dir=hyperspace > Hyperspace.Master.Host=192.168.2.69 > Hyperspace.Master.Port=38040 > Hyperspace.Master.Workers=20 > Hypertable.Master.Host=192.168.2.69 > Hypertable.Master.Port=38050 > Hypertable.Master.Workers=20 > Hypertable.RangeServer.Port=38060 > Hypertable.RangeServer.Reactors=2 > Hypertable.Request.Timeout=180000 > Hypertable.Verbose=true > ThriftBroker.Port=38080 > config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg > dfs-host=localhost > dfs-port=38030 > grace-period=200000 > hs-host=192.168.2.69 > hs-port=38040 > keepalive=30000 > lease-interval=1000000 > master-host=192.168.2.69 > master-port=38050 > pidfile=/opt/hypertable/0.9.2.7/run/Hypertable.RangeServer.pid > port=38060 > reactors=2 > timeout=180000 > verbose=true<http://0.9.2.7/run/Hypertable.RangeServer.pid%0Aport=38060%0Areactors=2%0Atimeout=180000%0Averbose=true> > 1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7- > alpha/src/cc/Hypertable/RangeServer/HyperspaceSessionHandler.cc:31) > Hyperspace session state change: SAFE > 1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7- > alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:292) log_dir=/ > hypertable/servers/192.168.2.69_38060/log > 1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7- > alpha/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:200) Event: > type=CONNECTION_ESTABLISHED from=192.168.2.69:38050 > 1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7- > alpha/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:201) Stopping > maintenance queue > 1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7- > alpha/src/cc/AsyncComm/ConnectionManager.cc:272) Event: > type=DISCONNECT from=192.168.2.69:38050; Problem connecting to Master, > will retry in 15000 milliseconds... > 1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7- > alpha/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:211) Event: > type=DISCONNECT from=192.168.2.69:38050 > 1260941870 ERROR Hypertable.RangeServer : local_recover (/root/src/ > hypertable-0.9.2.7-alpha/src/cc/Hypertable/RangeServer/RangeServer.cc: > 472): Hypertable::Exception: failed expectation: nread == > RSML_HEADER_SIZE - METALOG bad range server metalog header > at Hypertable::RangeServerMetaLogReader::RangeServerMetaLogReader > (Hypertable::Filesystem*, const Hypertable::String&) (/root/src/ > hypertable-0.9.2.7-alpha/src/cc/Hypertable/Lib/ > RangeServerMetaLogReader.cc:210) > > +++++++++++++++++++++++++++++++++++++++++++++++++++++ > Thanks, > Nilesh Katkar > > On Dec 16, 11:04 am, Doug Judd <[email protected]> wrote: > > Try running 'cap shell' and then executing the command: > > > > tail /opt/hypertable/0.9.2.7/log/Hypertable.RangeServer.log > > > > and post the output. > > > > - Doug > > > > On Tue, Dec 15, 2009 at 10:00 PM, Nilesh K <[email protected]> > wrote: > > > Hi, > > > Now we(Vivek,Me and Team) have installed NTP and successfully > > > synchronised clocks of connected machines. Also we are able to > > > distribute hypertable through cap dist. Now while trying to start, we > > > are getting message as "Waiting for Hypertable.RangeServer to come > > > up...". > > > > > Console has following output. > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > ++++++++++++++++++++++++++++++++++++++++++++++ > > > [r...@localhost conf]# cap start > > > * executing `start' > > > ** transaction: start > > > * executing `start_master' > > > * executing "/opt/hypertable/0.9.2.7/bin/start-dfsbroker.sh > > > hadoop --config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\ > > > \n /opt/hypertable/0.9.2.7/bin/start-hyperspace.sh --config=/ > > > opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/ > > > 0.9.2.7/bin/start-master.sh --config=/opt/hypertable/0.9.2.7/conf/ > > > hypertable.cfg;" > > > servers: > > > ["192.168.2.69"] > > > [192.168.2.69] executing > > > command > > > ** [out :: 192.168.2.69] DFS broker: available file descriptors: > > > 1024 > > > ** [out :: 192.168.2.69] DfsBroker.hadoop appears to be running > > > (20333): > > > ** [out :: 192.168.2.69] root 20333 20332 0 Dec15 pts/0 00:00:30 java > > > -classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/ > > > classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/ > > > test/classes:/opt/hypertable/ > 0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/ > > > lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/ > > > hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/ > > > lib/jetty-ext/*.jar< > http://0.9.2.7/build:/opt/hypertable/0.9.2.7/build/%0Atest/classes:/o...>org.hypertable.DfsBroker.hadoop.main > -- > > > verbose > > > ** [out :: 192.168.2.69] Hyperspace appears to be running > > > (20330): > > > ** [out :: 192.168.2.69] root 20330 20329 0 Dec15 pts/0 00:00:37 /opt/ > > > hypertable/0.9.2.7/bin/Hyperspace.Master --pidfile /opt/hypertable/ > > > 0.9.2.7/run/Hyperspace.pid --verbose > > > ** [out :: 192.168.2.69] Started > > > Hypertable.Master > > > command > > > finished > > > * executing > > > `start_slaves' > > > * executing "/opt/hypertable/0.9.2.7/bin/random-wait.sh 5 &&\\\n / > > > opt/hypertable/0.9.2.7/bin/start-dfsbroker.sh hadoop --config=/ > > > opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/ > > > 0.9.2.7/bin/start-rangeserver.sh --config=/opt/hypertable/ > > > 0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/ > 0.9.2.7/bin/start- > > > thriftbroker.sh --config=/opt/hypertable/0.9.2.7/conf/ > > > hypertable.cfg" > > > servers: ["192.168.2.69", "192.168.2.85", > > > "192.168.2.86"] > > > [192.168.2.69] executing > > > command > > > [192.168.2.86] executing > > > command > > > [192.168.2.85] executing > > > command > > > ** [out :: 192.168.2.69] DFS broker: available file descriptors: > > > 1024 > > > ** [out :: 192.168.2.69] DfsBroker.hadoop appears to be running > > > (20333): > > > ** [out :: 192.168.2.69] root 20333 20332 0 Dec15 pts/0 00:00:30 java > > > -classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/ > > > classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/ > > > test/classes:/opt/hypertable/ > 0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/ > > > lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/ > > > hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/ > > > lib/jetty-ext/*.jar< > http://0.9.2.7/build:/opt/hypertable/0.9.2.7/build/%0Atest/classes:/o...>org.hypertable.DfsBroker.hadoop.main > -- > > > verbose > > > ** [out :: 192.168.2.86] DFS broker: available file descriptors: > > > 1024 > > > ** [out :: 192.168.2.86] DfsBroker.hadoop appears to be running > > > (16585): > > > ** [out :: 192.168.2.86] root 16585 1 0 Dec15 ? 00:00:19 java - > > > classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/ > > > classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/ > > > test/classes:/opt/hypertable/ > 0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/ > > > lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/ > > > hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/ > > > lib/jetty-ext/*.jar< > http://0.9.2.7/build:/opt/hypertable/0.9.2.7/build/%0Atest/classes:/o...>org.hypertable.DfsBroker.hadoop.main > --verbose -- > > > config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg > > > ** [out :: 192.168.2.85] DFS broker: available file descriptors: > > > 1024 > > > ** [out :: 192.168.2.85] DfsBroker.hadoop appears to be running > > > (9748): > > > ** [out :: 192.168.2.85] root 9748 1 0 Dec15 ? 00:00:16 java - > > > classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/ > > > classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/ > > > test/classes:/opt/hypertable/ > 0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/ > > > lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/ > > > junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/ > > > hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/ > > > lib/jetty-ext/*.jar< > http://0.9.2.7/build:/opt/hypertable/0.9.2.7/build/%0Atest/classes:/o...>org.hypertable.DfsBroker.hadoop.main > --verbose -- > > > config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg > > > ** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come > > > up... > > > *** [err :: 192.168.2.86] /opt/hypertable/0.9.2.7/bin/ht-env.sh: line > > > 99: 19817 Segmentation fault (core dumped) $VALGRIND > > > $HYPERTABLE_HOME/bin/$servercmd --pidfile $pidfile "$@" >& > > > $logfile > > > ** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come > > > up... > > > ** [out :: 192.168.2.69] ERROR: Hypertable.RangeServer did not come > > > up > > > ** [out :: 192.168.2.86] ERROR: Hypertable.RangeServer did not come > > > up > > > ** [out :: 192.168.2.85] ERROR: Hypertable.RangeServer did not come > > > up > > > command finished > > > failed: "sh -c '/opt/hypertable/0.9.2.7/bin/random-wait.sh 5 &&\\\n > / > > > opt/hypertable/0.9.2.7/bin/start-dfsbroker.sh hadoop --config=/ > > > opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/ > > > 0.9.2.7/bin/start-rangeserver.sh > > > > ... > > > > read more ยป > > -- > > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<hypertable-dev%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > > > -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
