Hello There,
How are you ? I am trying to setup Hypertable on hadoop multi node.(I have
2 systems in my cluser named "master" and "slave")
When I run "*cap start*" command its gives the following output on terminal
root@CISM479:~# cap start
* executing `start'
** transaction: start
* executing `start_servers'
* executing `start_hyperspace'
* executing "/opt/hypertable/current/bin/start-hyperspace.sh
--config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg"
servers: ["master"]
[master] executing command
** [out :: master] Started Hyperspace
command finished in 5296ms
* executing `start_master'
* executing "/opt/hypertable/current/bin/start-dfsbroker.sh hadoop
--config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg &&\\\n
/opt/hypertable/current/bin/start-master.sh
--config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg &&\\\n
/opt/hypertable/current/bin/start-monitoring.sh"
servers: ["master"]
[master] executing command
** [out :: master] DFS broker: available file descriptors: 1024
** [out :: master] Started DFS Broker (hadoop)
** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come
up...
** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come
up...
** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come
up...
** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come
up...
** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come
up...
** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come
up...
** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come
up...
** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come
up...
** [out :: master] ERROR: Hypertable.Master did not come up
command finished in 133321ms
failed: "sh -c '/opt/hypertable/current/bin/start-dfsbroker.sh hadoop
--config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg &&\\\n
/opt/hypertable/current/bin/start-master.sh
--config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg &&\\\n
/opt/hypertable/current/bin/start-monitoring.sh'" on master
Can you please let me know the possible causes of the issue ? Below is the
content from log files
*Log file :DfsBroker.hadoop.log*
Num CPUs=2
HdfsBroker.Port=38030
HdfsBroker.Reactors=2
HdfsBroker.Workers=20
HdfsBroker.Hadoop.ConfDir=/usr/lib/hadoop-0.20/conf
Adding hadoop configuration file /usr/lib/hadoop-0.20/conf/hdfs-site.xml
Adding hadoop configuration file /usr/lib/hadoop-0.20/conf/core-site.xml
HdfsBroker.dfs.client.read.shortcircuit=false
HdfsBroker.dfs.replication=2
HdfsBroker.Server.fs.default.name=hdfs://master:54310
5 Mar, 2014 3:10:08 PM org.hypertable.AsyncComm.IOHandler DeliverEvent
INFO: [/192.168.2.225:56729 ; Wed Mar 05 15:10:08 IST 2014] Connection
Established
5 Mar, 2014 3:10:10 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler
handle
INFO: [/192.168.2.225:56729 ; Wed Mar 05 15:10:10 IST 2014] Disconnect -
COMM broken connection : Closing all open handles from /192.168.2.225:56729
Closed 0 input streams and 0 output streams for client connection
/192.168.2.225:56729
5 Mar, 2014 3:10:11 PM org.hypertable.AsyncComm.IOHandler DeliverEvent
INFO: [/192.168.2.225:57752 ; Wed Mar 05 15:10:11 IST 2014] Connection
Established
5 Mar, 2014 3:10:11 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler
handle
INFO: [/192.168.2.225:57752 ; Wed Mar 05 15:10:11 IST 2014] Disconnect -
COMM broken connection : Closing all open handles from /192.168.2.225:57752
Closed 0 input streams and 0 output streams for client connection
/192.168.2.225:57752
5 Mar, 2014 3:10:13 PM org.hypertable.AsyncComm.IOHandler DeliverEvent
INFO: [/192.168.2.225:57119 ; Wed Mar 05 15:10:13 IST 2014] Connection
Established
5 Mar, 2014 3:10:13 PM org.hypertable.AsyncComm.IOHandlerData
handle_message_body
WARNING: Received response for non-pending event
(id=0,version=1,total_len=38)
5 Mar, 2014 3:10:13 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Exists
INFO: Testing for existence of file '/hypertable/servers/master/log/mml
5 Mar, 2014 3:10:13 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Exists
INFO: Testing for existence of file '/hypertable/servers/master/log/mml
5 Mar, 2014 3:10:13 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Mkdirs
INFO: Making directory '/hypertable/servers/master/log/mml'
5 Mar, 2014 3:10:13 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Mkdirs
SEVERE: I/O exception while making directory
'/hypertable/servers/master/log/mml' -
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create
directory /hypertable/servers/master/log/mml. Name node is in safe mode.
Resources are low on NN. Safe mode must be turned off manually.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:2140)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2114)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:892)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1439)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1435)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1433)
5 Mar, 2014 3:10:13 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler
handle
INFO: [/192.168.2.225:57119 ; Wed Mar 05 15:10:13 IST 2014] Disconnect -
COMM broken connection : Closing all open handles from /192.168.2.225:57119
Closed 0 input streams and 0 output streams for client connection
/192.168.2.225:57119
*Log file : Hyperspace.log*
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Common/Config.cc:601) Initializing
Hyperspace.Master (Hypertable 0.9.7.16 (v0.9.7.16-0-g817a161))...
CPU cores count=2
CephBroker.MonAddr=10.0.1.245:6789
DfsBroker.Local.Root=fs/local
DfsBroker.Port=38030
HdfsBroker.Hadoop.ConfDir=/usr/lib/hadoop-0.20/conf
Hyperspace.GracePeriod=200000
Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.Replica.Dir=hyperspace
Hyperspace.Replica.Host=[master]
Hyperspace.Replica.Port=38040
Hyperspace.Replica.Reactors=2
Hypertable.Master.Port=38050
Hypertable.RangeServer.LowActivityPeriod=["* 2-4 * * *"]
Hypertable.RangeServer.Port=38060
Hypertable.Verbose=true
Qfs.MetaServer.Name=master
Qfs.MetaServer.Port=20000
ThriftBroker.Port=38080
config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg
dir=hyperspace
keepalive=30000
lease-interval=1000000
pidfile=/opt/hypertable/current/run/Hyperspace.pid
port=38040
reactors=2
verbose=true
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:157) BerkeleyDB base
directory = '/opt/hypertable/0.9.7.16/hyperspace'
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:111)
localhost=CISM479 localip=192.168.2.225
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:125)
Removing statedb
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:333)
Replication master init done
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff13c32b700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff13bb2a700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff13b329700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff13ab28700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff13a327700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff139b26700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff139325700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff138b24700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff138323700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff137b22700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff137321700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff136b20700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff13631f700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff135b1e700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff13531d700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff134b1c700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff13431b700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff133b1a700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff133319700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff132b18700
1394012400 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500)
Created DB handles for thread: 7ff142473740
1394012403 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:269) Create session for
192.168.2.225:59353
1394012403 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:281) created session 1
1394012403 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/request/RequestHandlerRenewSession.cc:74)
Session handle 1 created
1394012403 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event:
type=CONNECTION_ESTABLISHED from=192.168.2.225:57446
1394012403 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:343) Initialized session
1 (serverup)
1394012403 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/request/RequestHandlerDestroySession.cc:42)
Destroying session 1
1394012403 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:317) destroyed session
1(serverup)
1394012403 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:468) Expiring session 1
name=serverup
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:269) Create session for
192.168.2.225:40061
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:281) created session 2
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/request/RequestHandlerRenewSession.cc:74)
Session handle 2 created
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event:
type=CONNECTION_ESTABLISHED from=192.168.2.225:50565
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:343) Initialized session
2 (Hypertable.Master)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284)
exists(session_id=2, name=/hypertable)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting
exists(session_id=2, name=/hypertable)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284)
exists(session_id=2, name=/hypertable/master)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting
exists(session_id=2, name=/hypertable/master)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:1935) open(session_id=2,
session_name = Hypertable.Master, fname=/hypertable/master, flags=0x7,
event_mask=0x0)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2064) handle 1 created
('/hypertable/master', session=2(Hypertable.Master), flags=0x7, mask=0x0)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:703) exitting
open(session_id=2, session_name = Hypertable.Master,
fname=/hypertable/master, flags=0x7, event_mask=0x0)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:1237)
lock(session=2(Hypertable.Master), handle=1, mode=0x2, try_lock=1)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:1356) lock txn={BDbTxn
m_handle_namespace_db=0x16a25a0, m_handle_state_db=0x16a26c0, m_db_txn=0}
commited handle=1 node=/hypertable/master mode=2 status=1
lock_generation=3 notification_count=0
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2389)
attrexists(session=2(Hypertable.Master), handle=1, attr=next_server_id)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284)
exists(session_id=2, name=/hypertable/servers)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting
exists(session_id=2, name=/hypertable/servers)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284)
exists(session_id=2, name=/hypertable/tables)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting
exists(session_id=2, name=/hypertable/tables)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:1935) open(session_id=2,
session_name = Hypertable.Master, fname=/hypertable/root, flags=0xb,
event_mask=0x0)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2064) handle 2 created
('/hypertable/root', session=2(Hypertable.Master), flags=0xb, mask=0x0)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:703) exitting
open(session_id=2, session_name = Hypertable.Master,
fname=/hypertable/root, flags=0xb, event_mask=0x0)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:755)
close(session=2(Hypertable.Master), handle=2)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284)
exists(session_id=2, name=/hypertable/namemap/names)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting
exists(session_id=2, name=/hypertable/namemap/names)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284)
exists(session_id=2, name=/hypertable/namemap/ids)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting
exists(session_id=2, name=/hypertable/namemap/ids)
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/request/RequestHandlerDestroySession.cc:42)
Destroying session 2
1394012413 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:317) destroyed session
2(Hypertable.Master)
1394012414 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:468) Expiring session 2
name=Hypertable.Master
1394012414 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 1
1394012414 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:1541) Persisting lock
released notifications
1394012414 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:1552) Finished persisting
lock released notifications
*Log file : DfsBroker.local.log*FILE WAS EMPTY
*Log file : Hypertable.Master.log*1394012413 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Common/Config.cc:601) Initializing
Hypertable.Master (Hypertable 0.9.7.16 (v0.9.7.16-0-g817a161))...
CPU cores count=2
CephBroker.MonAddr=10.0.1.245:6789
DfsBroker.Local.Root=fs/local
DfsBroker.Port=38030
HdfsBroker.Hadoop.ConfDir=/usr/lib/hadoop-0.20/conf
Hyperspace.GracePeriod=200000
Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.Replica.Dir=hyperspace
Hyperspace.Replica.Host=[master]
Hyperspace.Replica.Port=38040
Hypertable.Master.Port=38050
Hypertable.Master.Reactors=2
Hypertable.RangeServer.LowActivityPeriod=["* 2-4 * * *"]
Hypertable.RangeServer.Port=38060
Hypertable.Verbose=true
Qfs.MetaServer.Name=master
Qfs.MetaServer.Port=20000
ThriftBroker.Port=38080
config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg
dfs-port=38030
grace-period=200000
hs-host=[master]
hs-port=38040
keepalive=30000
lease-interval=1000000
pidfile=/opt/hypertable/current/run/Hypertable.Master.pid
port=38050
reactors=2
verbose=true
1394012413 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/main.cc:434) Obtained lock
on '/hypertable/master'
1394012413 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:75)
rangeservers monitoring stats dir /opt/hypertable/0.9.7.16/run/monitoring
exists
1394012413 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:75)
rangeservers monitoring stats dir
/opt/hypertable/0.9.7.16/run/monitoring/tables exists
1394012413 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:75)
rangeservers monitoring stats dir
/opt/hypertable/0.9.7.16/run/monitoring/rangeservers exists
1394012413 ERROR Hypertable.Master : main
(/root/src/hypertable/src/cc/Hypertable/Master/main.cc:384):
Hypertable::Exception: Error mkdirs DFS directory
/hypertable/servers/master/log/mml - DFS BROKER i/o error
at virtual void Hypertable::DfsBroker::Client::mkdirs(const String&)
(/root/src/hypertable/src/cc/DfsBroker/Lib/Client.cc:517)
at virtual void Hypertable::DfsBroker::Client::mkdirs(const String&)
(/root/src/hypertable/src/cc/DfsBroker/Lib/Client.cc:514):
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create
directory /hypertable/servers/master/lo
Thank you.
--
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/hypertable-dev.
For more options, visit https://groups.google.com/groups/opt_out.
