Re: Re: HBase client shutdown hook hangs

Thanks St. Ack On Wed, 2010-10-20 at 20:22 -0700, Stack wrote: > On Tue, Oct 12, 2010 at 11:37 AM, Stack wrote: > > On Tue, Oct 12, 2010 at 2:27 PM, wrote: > >> It's all running fine and without exceptions before and after killing and > >> restarting. > >> > >> Every removed shutdown hook is

Re: Multiple HBase instances

Thanks Stack, We're using Hadoop/HBase for backend processing and storage and serving out of another database, so we don't have SLA for the response times of HBase but might for the run time of production jobs to be done by a give time. In this type of usage if we we're having around 40 nodes in

Re: The hfile.block.cache.size = 0 performance is better than default(0.2) in random read? Is it possible?

Hi Ryan, *hfile.block.cache.size = 0, GC log:* 2010-10-21T15:53:27.486+0800: 1428.317: [GC [PSYoungGen: 18270K->320K(17728K)] 62008K->44066K(61696K), *0.0043520 secs*] [Times: user=0.00 sys=0.00, real=0.00 secs] 2010-10-21T15:53:27.933+0800: 1428.764: [GC [PSYoungGen: 17641K->256K(17024K)] 61386K-

Hadoop and HBase Web UIs on EC2

Hi Thanks all for bailing me out time and again. I am using the HBase ec2 scripts to launch a cluster of machines on EC2 (hadoop 0.20.2 and hbase 0.20.6). I use my own AMI with hadoop and hbase downloaded to the paths as specified in the EC2 scripts. Hadoop and HBase work absolutely fine. However

Re: hive to hbase

I checked out hive trunk this week, it requires at least this patch to compile: http://pastebin.com/ZH8dQfd7 I haven't tested it out yet. J-D On Wed, Oct 20, 2010 at 8:55 PM, Jack Levin wrote: > Hello, does anyone have hbase_hive handler working for 0.89 Hbase > version?  We only have 0.20.6 ha

RE: The hfile.block.cache.size = 0 performance is better than default(0.2) in random read? Is it possible?

By using the block cache, read blocks are referenced within the block cache data structures and referenced for a longer amount of time than if not put into the block cache. This will definitely add additional stress to the GC. If you expect a very low hit ratio, it can be advantageous to not us

Re: hive to hbase

The attempt of running hive trunk against 0.89 causes masternotfound exception (at least yestarday did, there is an open Jira about it). -Jack On Thu, Oct 21, 2010 at 9:20 AM, Jean-Daniel Cryans wrote: > I checked out hive trunk this week, it requires at least this patch to > compile: http://pas

Re: hive to hbase

I don't see the jira Jack, could it be that your hive jobs aren't configured with zookeeper's addresses? Are the tasks trying to connect to localhost? J-D On Thu, Oct 21, 2010 at 9:50 AM, Jack Levin wrote: > The attempt of running hive trunk against 0.89 causes masternotfound > exception (at lea

master gets stuck at log replay

2010-10-21 10:08:14,268 WARN org.apache.hadoop.hbase.util.FSUtils: Waited 2014334ms for lease recovery on hdfs://namenode-rd.imageshack.us:9000/hbase/.logs/mtae6.prod.imageshack.com,60020,1287624295377/10.103.5.6%3A60020.1287672366636:org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: fa

Re: hive to hbase

We are running into the issue reported in HIVE-1597 (Hive CLI returns MasterNotRunningException with HBase 0.89.x) when using Hive from trunk and HBase 0.89.20100830, StumbleUpon branch. Does anyone here have possible workarounds for this issue, or does anyone successfully run Hive with HBase 0.8

Re: master gets stuck at log replay

This can happen when the original owner of the file is still alive, in your case is the region server it's recovering (10.103.5.6) is still running? If it GCed hard, then it probably stayed "alive" for a while but it should shut down when it wakes up. J-D On Thu, Oct 21, 2010 at 10:10 AM, Jack Le

Re: master gets stuck at log replay

How do we best resolve something like that, I just deleted that file... does it mean I might have lost inserts? -Jack On Thu, Oct 21, 2010 at 10:32 AM, Jean-Daniel Cryans wrote: > This can happen when the original owner of the file is still alive, in > your case is the region server it's recover

Re: master gets stuck at log replay

You should take a look at that server's region server process to see its health, it should recover (use jps to find if the process is still running, maybe tail the log to see what's going on, worst case you can kill -9). For how long was the master stuck? I remember there was an issue with for some

Re: Hadoop and HBase Web UIs on EC2

Are you using the external hostname or the internal one when trying to see the pages? AFAIK it doesn't bind on the external ones. Also, try to use "links" or any other text-based browser from the machines to see the web pages on their internal hostnames. J-D On Thu, Oct 21, 2010 at 8:12 AM, Kart

REST server disconnects

2010-10-21 10:15:09,845 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 1000ms, waiting for root region. 2010-10-21 10:15:10,846 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: locateRegionInMeta attempt 0 of 10 failed; retrying after sleep of 10

Re: hive to hbase

I think I found the issue, I commented in Hive's jira. Going to open one for HBase soon. J-D On Thu, Oct 21, 2010 at 10:13 AM, Jack Levin wrote: > We are running into the issue reported in HIVE-1597 (Hive CLI returns > MasterNotRunningException with HBase 0.89.x) when using Hive from > trunk and

Re: 0.89 Regionserver outage behaviors

Thanks St.Ack. So, looking at these configs, the default behavior should be to sync each individual entry directly to HDFS or at least synch to HDFS every second. Right? I ran this quick test: - 1 Master, 4 Regionservers all 5 on separate EC2 instances, default flushlogentries and optionall

something not right with ZK (was: Re: REST server disconnects)

[Changed the title of the mail, probably not a REST server issue.] This is the HBase client library embedded in the REST server warning that the znode in ZooKeeper corresponding to the root regionserver went away. It waits and waits for it to come back, but it never does. Nothing in the master