Hi,
I am using Hypertable 9.3.1 on ubuntu 8.04. A couple of days ago, I
could install and start the system perfectly. Then I shutdown the
nodes (with cap stop and hadoop$./stop-all.sh). When I tried to
restart, Hyperspace log at master is giving an error:
/-------------------
1275402797 ERROR Hyperspace.Master : run (/opt/hypertable/
hypertable-0.9.3.1-alpha/src/cc/Hyperspace/RequestHandlerOpen.cc:60):
Hypertable::Exception: node: '/hypertable/master' parent node: '/
hypertable' - HYPERSPACE file not found
at void
Hyperspace::Master::open(Hyperspace::ResponseCallbackOpen*, uint64_t,
const char*, uint32_t, uint32_t, std::vector<Hyperspace::Attribute,
std::allocator<Hyperspace::Attribute> >&) (/opt/hypertable/
hypertable-0.9.3.1-alpha/src/cc/Hyperspace/Master.cc:883)
--------------------/
/-------------------
$ ./ht hyperspace --exec "open /; readdir /;"
SESSION CALLBACK: Safe
Welcome to the hyperspace command interpreter.
For information about Hypertable, visit http://www.hypertable.org/
Type 'help' for a list of commands, or 'help shell' for a
list of shell meta commands.
(dir) hyperspace
-----------------------/
but I did create /hypertable directory
/---------------------
$ ./hadoop fs -ls /
Found 2 items
drwxrwxrwx - erts supergroup 0 2010-06-01 18:28 /hypertable
drwxr-xr-x - erts supergroup 0 2010-06-01 17:36 /mnt
-----------------------/
On May 29, 8:53 pm, Harshada <[email protected]> wrote:
> Oh.. sorry forgot to post here. Hypertable started successfully on
> hadoop. The problem was in my config file (hypertable.cfg). I changed
> localhost to master for Hypertable.Master.Host &
> Hyperspace.Replica.Host on slaves. All services are working fine.
>
> For my version of Hypertable there was no property like
> "Hyperspace.Master.Host" in hypertable.cfg, but there was one in
> Capfile "role :hyperspace".
>
> Thanks for your valuable inputs Sanjit.
>
> On May 29, 8:38 pm, Sanjit Jhala <[email protected]> wrote:
>
> > That error message shdn't be a problem. Can you try upgrading to 0.9.3.1 and
> > see if that fixes anything? Theres been many bugfixes since 0.9.2.8 and
> > though I can't think of one that explains this scenario, it would be easier
> > to debug/reproduce with the latest & greatest code.
>
> > So Hyperspace is up and running but the RangeServer can't connect to it? Can
> > you do a clean run and (cap cleandb; kill all dangling HT procs manually;
> > cap dist; cap start) and tar ball all the logs, Capfile and config
> > file here<http://groups.google.com/group/hypertable-user/files?pli=1>?
> > Btw I'm not a capistrano expert, but I wonder if the way you've defined the
> > roles is correct. If you look at the examples in the conf directory it looks
> > like it shd be:
> > *role :localhost, "master"*
>
> > Also I hope you have two servers named "master" & "slave" and you're running
> > the cap commands from "master". Btw do both RangeServers fail to come up or
> > just the one "slave" ?
> > If its just the one on "slave" can you try to connect to Hyperspace from
> > "slave" using the command line:
>
> > *<$INSTALL_DIR>/bin/ht hyperspace --conf <$HT_CONFIG_FILE>--exec "open /;
> > readdir /;"*
> > on "slave". If not I suspect you might have some firewall blocking access to
> > the Hyperspace port.
>
> > -Sanjit
>
> > On Sat, May 29, 2010 at 2:08 AM, Harshada Chavan
> > <[email protected]>wrote:
>
> > > $ps -ef | grep Hyperspace
>
> > > harshada 15228 1 0 14:20 ? 00:00:00
> > > /opt/hypertable/hypertable-0.9.2.8-alpha/0.9.2.8/bin/Hyperspace.Master--pidfile
> > > /opt/hypertable/hypertable-0.9.2.8-alpha/
> > > 0.9.2.8/run/Hyperspace.pid --verbose
> > > --config=/opt/hypertable/hypertable-0.9.2.8-alpha/
> > > 0.9.2.8/conf/hypertable.cfg
>
> > > ------------------------------------------------------------------------------------------------------
>
> > > Is Hyperspace running? Can you try something like
> > >> /opt/hypertable/hypertable-0.9.2.8-alpha/bin/ht hyperspace --exec "open
> > >> /;
> > >> readdir /;"
>
> > > When I ran above command on master, I got following output..
>
> > > -------------------------------------------------------------------------------
> > > $ ./ht hyperspace --exec "open /; readdir /;"
> > > SESSION CALLBACK: Safe
>
> > > Welcome to the hyperspace command interpreter.
> > > For information about Hypertable, visithttp://www.hypertable.org/
>
> > > Type 'help' for a list of commands, or 'help shell' for a
> > > list of shell meta commands.
>
> > > (dir) hyperspace
> > > (dir) hypertable
>
> > > ---------------------------------------------------------------------------------
>
> > > seems it's running. But then why RangeServer log at slave says,
>
> > > --------------------
>
> > > "1275077211 ERROR Hypertable.RangeServer :
> > > (/opt/hypertable/hypertable-0.9.2.8-alpha/src/cc/Hypertable/RangeServer/main.cc:86)Unable
> > > to connect to hyperspace, exiting..."
> > > --------------------
>
> > > Also, in log/Hyperspace.log at master I see following error:
>
> > > ------------------------------
> > > 1275123032 ERROR Hyperspace.Master : run
> > > (/opt/hypertable/hypertable-0.9.2.8-alpha/src/cc/Hyperspace/RequestHandlerOpen.cc:60):
> > > Hypertable::Exception: node: '/hypertable/master' parent node:* *
> > > '/hypertable' - HYPERSPACE file not found **
>
> > > at void
> > > Hyperspace::Master::open(Hyperspace::ResponseCallbackOpen*,
> > > uint64_t, const char*, uint32_t, uint32_t,
> > > std::vector<Hyperspace::Attribute, std::allocator<Hyperspace::Attribute>
> > > >&)
> > > (/opt/hypertable/hypertable-0.9.2.8-alpha/src/cc/Hyperspace/Master.cc:877)
> > > --------------------
>
> > >> -Sanjit
>
> > >> On Fri, May 28, 2010 at 2:32 PM, Harshada
> > >> <[email protected]>wrote:
>
> > >>> On May 29, 2:10 am, Sanjit Jhala <[email protected]> wrote:
> > >>> > Hi Harshada,
>
> > >>> > I'm confused about what state your setup is in.
>
> > >>> I have followed procedure given at
> > >>>http://code.google.com/p/hypertable/wiki/UpAndRunningWithHadoop
> > >>> and have been successful till "cap dist".
>
> > >>> >Can you try the following.
> > >>> > 1. cap cleandb
> > >>> > 2. Run ps and make sure all Hypertable related procs (Hyperspace,
> > >>> > RangeServer, Master, DFSBroker, ThroftBroker, any dangling start/stop
> > >>> > scripts) have been stopped. If not kill them manually.
>
> > >>> Killed everything.
>
> > >>> > 3. just to be sure run cap cleandb again
>
> > >>> Executed this command again. It's output is exactly similar to what I
> > >>> have posted in the reply to your earlier reply (my prev to prev mail).
>
> > >>> > 4. cap start
>
> > >>> Again same output as posted earlier in the same mail.
>
> > >>> > Also are you running on a single node? If not is your capfile updated
> > >>> with
> > >>> > all the servers in your cluster?
>
> > >>> my Capfile:
>
> > >>> set :source_machine, "master"
> > >>> set :install_dir, "/opt/hypertable/hypertable-0.9.2.8-alpha"
> > >>> set :hypertable_version, "0.9.2.8"
> > >>> set :default_dfs, "hadoop"
> > >>> set :default_config, "/opt/hypertable/hypertable.cfg"
>
> > >>> role :master, "master"
> > >>> role :hyperspace, "master"
> > >>> role :slave, "master", "slave"
> > >>> role :localhost, "127.0.0.1"
>
> > >>> and hypertable.cfg:
>
> > >>> #
> > >>> # hypertable.cfg
> > >>> #
>
> > >>> # HDFS Broker
> > >>> HdfsBroker.Port=38030
> > >>> HdfsBroker.fs.default.name=hdfs://master:54310
> > >>> HdfsBroker.Workers=20
>
> > >>> # Ceph Broker
> > >>> CephBroker.Port=38030
> > >>> CephBroker.Workers=20
> > >>> CephBroker.MonAddr=10.0.1.245:6789
>
> > >>> # Local Broker
> > >>> DfsBroker.Local.Port=38030
> > >>> DfsBroker.Local.Root=fs/local
>
> > >>> # DFS Broker - for clients
> > >>> DfsBroker.Host=localhost
> > >>> DfsBroker.Port=38030
>
> > >>> # Hyperspace
> > >>> Hyperspace.Replica.Host=localhost
> > >>> Hyperspace.Replica.Port=38040
> > >>> Hyperspace.Replica.Dir=hyperspace
> > >>> Hyperspace.Replica.Workers=20
>
> > >>> # Hypertable.Master
> > >>> Hypertable.Master.Host=localhost
> > >>> Hypertable.Master.Port=38050
> > >>> Hypertable.Master.Workers=20
>
> > >>> # Hypertable.RangeServer
> > >>> Hypertable.RangeServer.Port=38060
>
> > >>> Hyperspace.KeepAlive.Interval=30000
> > >>> Hyperspace.Lease.Interval=1000000
> > >>> Hyperspace.GracePeriod=200000
>
> > >>> # ThriftBroker
> > >>> ThriftBroker.Port=38080
>
> > >>> **NOTE: There's no property like: Hyperspace.Master.Host. Looking at
> > >>> the log files, it takes "localhost" as the value.
>
> > >>> After reading log files (as posted in my prev mail), I think the
> > >>> problem is with Hyperspace not being up at master. Because the log for
> > >>> RangServer at Slave machine says:
>
> > >>> "1275082230 ERROR Hypertable.RangeServer : (/opt/hypertable/
> > >>> hypertable-0.9.2.8-alpha/src/cc/Hypertable/RangeServer/main.cc:86)
> > >>> Unable to connect to hyperspace, exiting..."
>
> > >>> > -Sanjit
>
> > >>> > On Fri, May 28, 2010 at 1:43 PM, Harshada <[email protected]>
> > >>> wrote:
> > >>> > > Looks like this issue is related to the Hyperspace @master.
>
> > >>> > > When I checked RangServer's log at slave, I found this
>
> > >>> > > "1275077211 ERROR Hypertable.RangeServer : (/opt/hypertable/
> > >>> > > hypertable-0.9.2.8-alpha/src/cc/Hypertable/RangeServer/main.cc:86)
> > >>> > > Unable to connect to hyperspace, exiting..."
>
> > >>> > > I checked Hyperspace's log at master for the error:
>
> > >>> > > 1275076989 ERROR Hyperspace.Master : run (/opt/hypertable/
> > >>> > > hypertable-0.9.2.8-alpha/src/cc/Hyperspace/RequestHandlerOpen.cc:60):
> > >>> > > Hypertable::Exception: node: '/hypertable/master' parent node: '/
> > >>> > > hypertable' - HYPERSPACE file not found
> > >>> > > at void
> > >>> > > Hyperspace::Master::open(Hyperspace::ResponseCallbackOpen*,
> > >>> > > uint64_t,
> > >>> > > const char*, uint32_t, uint32_t, std::vector<Hyperspace::Attribute,
> > >>> > > std::allocator<Hyperspace::Attribute> >&) (/opt/hypertable/
> > >>> > > hypertable-0.9.2.8-alpha/src/cc/Hyperspace/Master.cc:877)
>
> > >>> > > What does HYPERSPACE file not found mean?
>
> > >>> > > There exists a directory /hypertable at HDFS at the master with 777
> > >>> > > permission.
>
> > >>> > > Please help.
>
> > >>> > > On May 29, 12:00 am, Harshada <[email protected]> wrote:
> > >>> > > > Thanks for your reply Sanjit.
>
> > >>> > > > 'cap cleandb' gives following output: (with one error at the end)
>
> > >>> > > > * executing `stop'
> > >>> > > > ** transaction: start
> > >>> > > > * executing `stop_slaves'
> > >>> > > > * executing "/opt/hypertable/hypertable-0.9.2.8-alpha/
> > >>> 0.9.2.8/bin/
> > >>> > > > stop-servers.sh --no-hyperspace"
> > >>> > > > servers: ["master", "slave"]
> > >>> > > > [master] executing command
> > >>> > > > ** [out :: master] Sending shutdown command
> > >>> > > > ** [out :: master] Unable to establish connection to range server
> > >>> > > > [slave] executing command
> > >>> > > > ** [out :: slave] Sending shutdown command
> > >>> > > > ** [out :: slave] Unable to establish connection to range server
> > >>> > > > ** [out :: master] Shutdown range server complete
> > >>> > > > ** [out :: slave] Shutdown range server complete
> > >>> > > > ** [out :: master] Shutdown DFS broker complete
> > >>> > > > ** [out :: master] Shutdown thrift broker complete
> > >>> > > > ** [out :: master] Shutdown hypertable master complete
> > >>> > > > ** [out :: slave] Shutdown thrift broker complete
> > >>> > > > ** [out :: slave]
>
> ...
>
> read more »
--
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en.
