Re: Of hbase key distribution and query scalability, again.

of some > external caching mechanism) that'd alleviate this; at base, you still have to > ask exactly one RS for any given piece of data. > > Ian > > On May 25, 2012, at 12:31 PM, Dmitriy Lyubimov wrote: > >> Hello, >> >> I'd like to collect opin

Of hbase key distribution and query scalability, again.

Hello, I'd like to collect opinions from HBase experts on the query uniformity and whether there's any advance technique currently exists in HBase to cope with the problems of query uniformity beyond just maintaining the key uniform distribution. I know we start with the statement that in order t

Re: How to define a custom filter to skip some amount of rows?

Filters support re-seek functionality. Hint can be given on the point of re-seek. Re-seek can be applied to either columns or rows (basically, hint is the new target of re-seek as key-value). You can find details in the hbase book (and perhaps somewhere online, too). -d On Mon, Feb 13, 2012 at 5:

Re: HBase and R

There's something for hbase in rhadoop project I think. But I haven't tried it myself, yet anyway. On Feb 1, 2012 4:21 AM, "Peter Wolf" wrote: > Anyone out there using R with HBase? > > I found HBaseR > > https://bitbucket.org/jspies/**hbaser/src/cc8bf79e8d99/hbase.**r

Re: Table region got stuck, doesn't move/assign

e our hbase stack in prod. Thank you very much, sir. -d On Wed, Jan 18, 2012 at 9:34 AM, Stack wrote: > On Mon, Jan 16, 2012 at 3:45 PM, Dmitriy Lyubimov wrote: >> i have a table which seems to get stuck in a state where it can't be >> queried, moved or split/compacted. >

Re: Table region got stuck, doesn't move/assign

Another weirdness, where it all started while that regions was still where it was, was that any attempt to query that table caused indefinite hanging (call never returned). Now of course that we killed/restarted the RS it just says 'region not serving'. On Mon, Jan 16, 2012 at 3:45 P

Table region got stuck, doesn't move/assign

Hi, i have a table which seems to get stuck in a state where it can't be queried, moved or split/compacted. The logs don't have any error statements. Our admin tried hbck to no avail . We stopped the region server, table did not get reassigned. (all other did). when bround in UI, this table just

Re: HBase for ad-hoc aggregate queries

series data feeds. -d On Wed, Jan 11, 2012 at 11:48 AM, Dmitriy Lyubimov wrote: > IMO You will never get the same flexibility. There are also numerous > differences in data modelling approach (TTL, uniformly-distributed ids > requirement to scale query volume, etc.) > > The most

Re: HBase for ad-hoc aggregate queries

IMO You will never get the same flexibility. There are also numerous differences in data modelling approach (TTL, uniformly-distributed ids requirement to scale query volume, etc.) The most flexibility in that regard we reached so far w.r.t. aggregation queries is OLAPish model (see link on HBase

Re: Strategies for aggregating data in a HBase table

help there). On Wed, Dec 21, 2011 at 12:14 AM, Dmitriy Lyubimov wrote: > https://github.com/dlyubimov/HBase-Lattice > > On Wed, Dec 21, 2011 at 12:13 AM, Dmitriy Lyubimov wrote: >> Thomas, >> >> Sorry for shameless self-promotion. Can you look at our hbase-lattice >>

Re: Strategies for aggregating data in a HBase table

https://github.com/dlyubimov/HBase-Lattice On Wed, Dec 21, 2011 at 12:13 AM, Dmitriy Lyubimov wrote: > Thomas, > > Sorry for shameless self-promotion. Can you look at our hbase-lattice > project? it is incremental OLAP-ish cube compilation with custom > filtering to optimize fo

Re: Strategies for aggregating data in a HBase table

Thomas, Sorry for shameless self-promotion. Can you look at our hbase-lattice project? it is incremental OLAP-ish cube compilation with custom filtering to optimize for composite key scans. Some rudimental query language as well. Bunch of standard (and not so standard) aggregates for measure data

OLAP-ish incremental BI capabilities for hbase, looking for collaborators.

Hello, We (Inadco) are looking for users and developers to engage in our open project code named, for the lack of better name, "hbase-lattice" in order to mutually benefit and eventually develop a mature Hbase-based BI real time OLAP-ish solution. The basic premise is to use Cuboid Lattice -like

Re: Hbase Master Failover Issue

Sat, May 14, 2011 at 3:04 PM, Stack wrote: > What did you do to solve it? > Thanks, > St.Ack > > On Fri, May 13, 2011 at 6:17 PM, Dmitriy Lyubimov wrote: >> Ok i think the issue is largely solved. Thanks for your help, guys. >> >> -d >> >> On Fri, May

Re: Hbase Master Failover Issue

Ok i think the issue is largely solved. Thanks for your help, guys. -d On Fri, May 13, 2011 at 5:32 PM, Dmitriy Lyubimov wrote: > ok the problem seems to be multi-nic hosting on masters. the hbase > master starts up and uses canonical hostname to listen on which points > to a wrong

Re: Hbase Master Failover Issue

o use it with . thanks. -d On Fri, May 13, 2011 at 3:39 PM, Dmitriy Lyubimov wrote: > Thanks, Jean-Daniel. > > Logs don't show anything abnormal (not even warnings). How soon you > think the region servers should join? > > I am guessing the sequence should be something alo

Re: Hbase Master Failover Issue

Thanks, Jean-Daniel. Logs don't show anything abnormal (not even warnings). How soon you think the region servers should join? I am guessing the sequence should be something along the lines -- zookeeper needs to timeout old master session first (2 mins or so ) , then hot spare should wean next m

Re: RPC metrics coming up as 0

thanks i already did that :) On Thu, Apr 21, 2011 at 10:50 PM, Stack wrote: > On Thu, Apr 21, 2011 at 12:52 AM, Dmitriy Lyubimov wrote: >> On a completely other issue i was reporting, I still have no ideas why >> remote client latencies would hover around 25ms in this case if t

Re: 0.90 latency performance, cdh3b4

Thank you, sir. On Fri, Apr 22, 2011 at 12:31 PM, tsuna wrote: > On Fri, Apr 22, 2011 at 12:15 PM, Dmitriy Lyubimov wrote: >> is it possible to configure this client to open more than one socket >> connection from same client to same region server? >> In other words, is H

Re: 0.90 latency performance, cdh3b4

Benoit, Thank you. is it possible to configure this client to open more than one socket connection from same client to same region server? In other words, is HBASE-2939 already non-issue there? > asynchbase implements the HBase RPC protocol in a different way, it's > written from scratch.  It us

Re: 0.90 latency performance, cdh3b4

11 at 12:03 PM, Jean-Daniel Cryans wrote: > It's all multiplexed. > > J-D > > On Fri, Apr 22, 2011 at 11:52 AM, Dmitriy Lyubimov wrote: >>> >>> I doubt that TCP doesn't perform well.  If you really believe so, can >>> you provide a packet captu

Re: 0.90 latency performance, cdh3b4

> > I doubt that TCP doesn't perform well.  If you really believe so, can > you provide a packet capture collected with: > sudo tcpdump -nvi eth0 -s0 -w /tmp/pcap port 60020 > Hm. What i discovered there is that I assumed my hack at RS connection pooling was working but it doesn't seem to be. Even

Re: 0.90 latency performance, cdh3b4

> > You saw "HBASE-2939  Allow Client-Side Connection Pooling"?  Would that help? Ok just read thru the issue. That's exactly what i thought upon reading the code in HBaseClient class. Although in my cluster it did not seem to have more than about 20% effect and it was more or less evaporated afte

Re: 0.90 latency performance, cdh3b4

s (not a > benefit here) > > But getting this to work was weeks of work and months of testing with > thousands of different clients.  I wouldn't want to repeat that without > serious reasons. > > > On Thu, Apr 21, 2011 at 11:12 PM, Dmitriy Lyubimov wrote: > >> > Wh

Re: 0.90 latency performance, cdh3b4

> I doubt that TCP doesn't perform well.  If you really believe so, can > you provide a packet capture collected with: > sudo tcpdump -nvi eth0 -s0 -w /tmp/pcap port 60020 Thanks, i will certainly try. However same class machine same data same test locally vs. remote same subnet is de facto 100% d

Re: 0.90 latency performance, cdh3b4

nd sometimes near > 1ms. > > My first suspicion would be a concurrency limit somewhere that is > artificially throttling things down.  Why it would be sooo extreme, I cannot > imagine. > > On Thu, Apr 21, 2011 at 10:49 PM, Dmitriy Lyubimov wrote: > >> So of course this test i

Re: 0.90 latency performance, cdh3b4

, Ted Dunning wrote: > Dmitriy, > > Did I hear you say that you are instantiating a new Htable for each request? >  Or was that somebody else? > > On Thu, Apr 21, 2011 at 11:04 PM, Stack wrote: > >> On Thu, Apr 21, 2011 at 10:49 PM, Dmitriy Lyubimov >> wrote: >>

Re: 0.90 latency performance, cdh3b4

yes this is for 500 QPS of scans returning back approx. 15k worth of data total. > > You saw "HBASE-2939  Allow Client-Side Connection Pooling"?  Would that help? Interesting. let me take a look. i kind of was thinking maybe there's some sense to allow to pool more than one tcp connection from sam

Re: 0.90 latency performance, cdh3b4

that using udp for short scans and gets might reduce latency a bit as well. On Wed, Apr 20, 2011 at 3:05 PM, Dmitriy Lyubimov wrote: > So i can't seem to be able to immediately find the explanation for those > metrics > > - rpcQueueTime -- do I assume it correctly it's the

Re: RPC metrics coming up as 0

hat was the solution? > St.Ack > > On Wed, Apr 20, 2011 at 11:15 PM, Dmitriy Lyubimov > wrote: >> ok, i think i solved the mistery. thanks for help. >> >

RPC metrics coming up as 0

Hi, so i enabled metrics with JConsole as described in the metrics page. (well actually some names have changed in hbase-env.sh so it's not verbatim). So what i am seeing is that for rpc metrics on a regionserver, only maxTime metric seems to change for rpcProcessingTime and rpcQueueTime. The res

Re: RPC metrics coming up as 0

PS. I can kind of beleive that avg time is 0, but numOps surely can't be 0? On Wed, Apr 20, 2011 at 5:39 PM, Dmitriy Lyubimov wrote: > Hi, > > so i enabled metrics with JConsole as described in the metrics page. > (well actually some names have changed in hbase-env.sh so it

Re: RPC metrics coming up as 0

I also saw several calls statistics where min=max>0 but average and num ops are 0. Can't be. Min and Max seem to be working but not averages and num ops. Any idea why it might be? Thanks. On Wed, Apr 20, 2011 at 6:04 PM, Dmitriy Lyubimov wrote: > PS. I can kind of beleive that av

Re: RPC metrics coming up as 0

ok, i think i solved the mistery. thanks for help.

Re: Latency related configs for 0.90

2011 at 3:04 PM, Dmitriy Lyubimov wrote: > >> exactly my symptoms with 0.90.1. What gives. >> >> On Wed, Apr 20, 2011 at 9:54 AM, George P. Stathis >> wrote: >> > Ted, what makes you say that? Have you seen similar issues in >> > pseudo-clu

Re: 0.90 latency performance, cdh3b4

time of request being processed by region server's handler? So inner time to last byte should be approximately sum of those, right? Thanks. -Dmitriy On Wed, Apr 20, 2011 at 1:17 PM, Dmitriy Lyubimov wrote: > Yes that's what i said. there's metric for fs latency but we are no

Re: 0.90 latency performance, cdh3b4

btw, Ted, your version of YCSB in github should show TTLBs, right? On Wed, Apr 20, 2011 at 1:14 PM, Ted Dunning wrote: > FS latency shouldn't matter with your 99.9% cache hit rate as reported. > > On Wed, Apr 20, 2011 at 12:55 PM, Dmitriy Lyubimov wrote: > >> Yes -

Re: 0.90 latency performance, cdh3b4

er with your 99.9% cache hit rate as reported. > > On Wed, Apr 20, 2011 at 12:55 PM, Dmitriy Lyubimov wrote: > >> Yes -- I already looked thru 'regionserver' metrics some time ago in >> hbase book. And i am not sure there's a 'inner ttlb' metric. >>

Re: 0.90 latency performance, cdh3b4

on hbase home page. > > On Wed, Apr 20, 2011 at 12:44 PM, Dmitriy Lyubimov wrote: >> Is there any way to log 'inner' TTLB times the region server incurs for >> reads? >> >> >> On Wed, Apr 20, 2011 at 12:43 PM, Dmitriy Lyubimov wrote: >>> i

Re: 0.90 latency performance, cdh3b4

Is there any way to log 'inner' TTLB times the region server incurs for reads? On Wed, Apr 20, 2011 at 12:43 PM, Dmitriy Lyubimov wrote: > i just enabled debug logging for o.a.h.hbase logger in that particular > region server... so far not much except for LRUBlock cache sp

Re: 0.90 latency performance, cdh3b4

gle regionserver.  Tail > that regionservers logs.  It might tell us something. > St.Ack > > On Wed, Apr 20, 2011 at 12:25 PM, Stack wrote: >> On Wed, Apr 20, 2011 at 12:25 PM, Stack wrote: >>> On Tue, Apr 19, 2011 at 4:46 PM, Dmitriy Lyubimov >>> wrote: >&

Re: 0.90 latency performance, cdh3b4

, hits=55732083, hitRatio=99.99%%, cachingAccesses=55732195, cachingHits=55732083, cachingHitsRatio=99.99%%, evictions=0, evicted=0, evictedPerRun=NaN On Wed, Apr 20, 2011 at 12:25 PM, Stack wrote: > On Wed, Apr 20, 2011 at 12:25 PM, Stack wrote: >> On Tue, Apr 19, 2011 at 4:46 PM

Re: 0.90 latency performance, cdh3b4

8% percentile. sorry. 8% requests do land under 3ms or less. On Wed, Apr 20, 2011 at 12:06 PM, Ted Dunning wrote: > What is meant by 8% quartile?  75th %-ile?  98%-ile?  Should quartile have > been quantile? > > On Wed, Apr 20, 2011 at 12:00 PM, Dmitriy Lyubimov wrote: > >&

Re: Latency related configs for 0.90

exactly my symptoms with 0.90.1. What gives. On Wed, Apr 20, 2011 at 9:54 AM, George P. Stathis wrote: > Ted, what makes you say that? Have you seen similar issues in > pseudo-clustered mode? We have been running in that mode on our dev > environment for a year now, we haven't had any issues like

Re: 0.90 latency performance, cdh3b4

10:06 AM, Dmitriy Lyubimov wrote: > Got it. This must be the reason. Cause it is a laugh check, and i do > see 6 regions for 40 rows so it can span them, although i can't > confirm it for sure. It may be due to how table was set up or due to > some time running them and rotatin

Re: 0.90 latency performance, cdh3b4

table with one region to get back to 1 region? or maybe there's a way to get it back to one region without recreating it, such as major compaction? thanks. -d On Wed, Apr 20, 2011 at 9:55 AM, Stack wrote: > On Wed, Apr 20, 2011 at 9:49 AM, Dmitriy Lyubimov wrote: >> Ok. Let me as

Re: 0.90 latency performance, cdh3b4

> On Tue, Apr 19, 2011 at 5:39 PM, Dmitriy Lyubimov wrote: >> PS so what should latency be for reads in 0.90, assuming moderate thruput? >> >> On Tue, Apr 19, 2011 at 5:39 PM, Dmitriy Lyubimov wrote: >>> for this test, there's just no more than 40 rows in every

Re: 0.90 latency performance, cdh3b4

evity. Sent from my android. -Dmitriy On Apr 19, 2011 6:29 PM, "Ted Dunning" wrote: > For a tiny test like this, everything should be in memory and latency > should be very low. > > On Tue, Apr 19, 2011 at 5:39 PM, Dmitriy Lyubimov wrote: >> PS so what should laten

Re: 0.90 latency performance, cdh3b4

also we had another cluster running previous CDH versions with pre-0.89 hbase and the latencies weren't as nearly as bad. On Tue, Apr 19, 2011 at 5:39 PM, Dmitriy Lyubimov wrote: > PS so what should latency be for reads in 0.90, assuming moderate thruput? > > On Tue, Apr 19, 2

Re: 0.90 latency performance, cdh3b4

PS so what should latency be for reads in 0.90, assuming moderate thruput? On Tue, Apr 19, 2011 at 5:39 PM, Dmitriy Lyubimov wrote: > for this test, there's just no more than 40 rows in every given table. > This is just a laugh check. > > so i think it's safe to assume it a

Re: 0.90 latency performance, cdh3b4

for this test, there's just no more than 40 rows in every given table. This is just a laugh check. so i think it's safe to assume it all goes to same region server. But latency would not depend on which server call is going to, would it? Only throughput would, assuming we are not overloading. An

0.90 latency performance, cdh3b4

Hi, I would like to see how i can attack hbase performance. Right now i am shooting scans returning between 3 and 40 rows and regardless of data size, approximately 500-400 QPS. The data tables are almost empty and in-memory, so they surely should fit in those 40% heap dedicated to them. My loca

Re: Zookeeper connection error on mapreduce HBase writes

This stack trace really starts getting familiar. Check next email threads, it is likely caused by zookeeper connection starvation. Of course moving htable to setup method is the right call. But i would also like to point out that if you are using hbase 0.90, each new instance of Htable(conf) will

Re: zookeeper-3.3.2 has default maxClientCnxns set to 10?

or perhaps scratch that. it seems you are saying the problem arises on the backend and your reducer code certainly doesn't create over 10 connections there. so it might be a combination of other tasks running at the same address. On Sat, Mar 26, 2011 at 2:04 AM, Dmitriy Lyubimov wrote: &g

Re: zookeeper-3.3.2 has default maxClientCnxns set to 10?

yes i had a very similar issue although i prefer to think about it in terms of hbase and by extension zk connection leak in TableInputFormat rather than adjusting max zk connection to 30 'cause sooner or later you will run out of it too. The problem is that HConnectionManager now identifies hbase

Re: zk connection leak with TableInput/OutputFormat (CDH3b4, 0.90.1)

s second parameter to deleteConnection(). > > On Wed, Mar 23, 2011 at 1:54 PM, Dmitriy Lyubimov wrote: > >> Hi, >> >> I am experiencing severe connection leak in my MR client that uses >> Hbase as input/output . Every job that uses TableInputFormat leaks 1 >> zooke

zk connection leak with TableInput/OutputFormat (CDH3b4, 0.90.1)

Hi, I am experiencing severe connection leak in my MR client that uses Hbase as input/output . Every job that uses TableInputFormat leaks 1 zookeeper connection per run as evidenced by netstat. I understand that the way HTable manages connections now is it creates a new HBase (and also Zookeeper)

Re: table creation is failing now and then (CDH3b3)

Thank you, sir. On Thu, Feb 24, 2011 at 4:12 PM, Jean-Daniel Cryans wrote: > See http://hbase.apache.org/upgrading.html#upgrade0.90 > > Also if you want to stay on CDH, CDH3b4 was just released. > > J-D > > On Thu, Feb 24, 2011 at 4:08 PM, Dmitriy Lyubimov > w

Re: table creation is failing now and then (CDH3b3)

, a lot of these kinds of > issues were fixed. > > -ryan > > On Wed, Feb 23, 2011 at 12:02 PM, Dmitriy Lyubimov > wrote: > > Hi all, > > > > from time to time we come to a sitation where .META. table seems to be > stuck > > in some corrupted state. > &g

table creation is failing now and then (CDH3b3)

Hi all, from time to time we come to a sitation where .META. table seems to be stuck in some corrupted state. In particular, attempts to create more tables cause ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in .META. for region LEAD_DATA,,129848469894

Re: Representing Hierarchy in HBase

I guess you might potentially have saved on key len building a secondary nested sets index or something along those lines as in http://en.wikipedia.org/wiki/Nested_set_model but simplicity of key path approach is hard to beat of course. On Mon, Feb 21, 2011 at 10:46 AM, Vishal Kapoor wrote: > Fri

Re: Command line integration question

Yes it was meant to be mahout's. Honest error, sorry. apologies for brevity. Sent from my android. -Dmitriy On Dec 6, 2010 3:08 AM, "Lars George" wrote:

Re: Command line integration question

Ok, i think i got it. Mahout uses standard ToolRunner to preconfigure the client. Got it. Thanks. On Sun, Dec 5, 2010 at 3:28 PM, Dmitriy Lyubimov wrote: > PS. also if I needed to play with various MR settings, such as child > processes arguments, could i pass that on to Configuration

Re: Command line integration question

ou in advance. -Dmitriy On Sun, Dec 5, 2010 at 3:17 PM, Dmitriy Lyubimov wrote: > Dear all, > > I am testing the command line integration for the SSVD patch in hadoop mode > and running into some difficulties. > Even that I defined $HADDOP_HOME and $HADOOP_CONF_DIR, apparently df

Command line integration question

Dear all, I am testing the command line integration for the SSVD patch in hadoop mode and running into some difficulties. Even that I defined $HADDOP_HOME and $HADOOP_CONF_DIR, apparently dfs configuration is not being picked up. I do run on CDH3b3, however, all hadoop configuration is 100% compa

Re: Data migration 0.20 -> 0.89 Q

itriy On Thu, Sep 30, 2010 at 11:31 AM, Jean-Daniel Cryans wrote: > Sorry, no migration as in there's nothing special to do... it just > works. And like Ryan said, no coming back. > > J-D > > On Thu, Sep 30, 2010 at 11:30 AM, Dmitriy Lyubimov > wrote: >

Re: Data migration 0.20 -> 0.89 Q

there are non-backward compatible > changes in how we name region directories, so the first split you have > will create region directories that 0.20 wont know about. > > -ryan > > On Thu, Sep 30, 2010 at 11:24 AM, Dmitriy Lyubimov > wrote: > > Hi, > > > > I trie

Re: Data migration 0.20 -> 0.89 Q

Are at least export file format compatible? On Thu, Sep 30, 2010 at 11:28 AM, Jean-Daniel Cryans wrote: > There's no migration. > > J-D > > On Thu, Sep 30, 2010 at 11:24 AM, Dmitriy Lyubimov > wrote: > > Hi, > > > > I tried to find the info on da

Data migration 0.20 -> 0.89 Q

Hi, I tried to find the info on data migration from 0.20 to 0.89 and so far wasn't successful. I noticed that migrate utility is not in the code base for the 0.89 anymore. is there a way or howto somewhere about how to migrate compacted 0.20 tables to 0.89 tables? Thanks in advance . Dmitriy

Re: Upgrading 0.20.6 -> 0.89

ime out and the master to presume the RS is dead and recover the > log. At this point when the RS recovers it must self terminate > because the regions dont 'belong' to itself anymore. > > > > On Wed, Sep 29, 2010 at 10:51 AM, Dmitriy Lyubimov > wrote: > >>

Re: Upgrading 0.20.6 -> 0.89

> Full GCs do happen. We have it at 40 seconds here. Jean-Daniel, Is it total with concurrent CMS? 40 seconds is a plausible number for full CMS, even more plausible for i-CMS, so i assume that's what you are quoting here. But CMS doesn't pause jvm for that much time. Most of that time is spent

Re: WAL, failover questions

St. Ack, just to make sure: by cdh3b2 aka 'hadoop-append' you imply hadoop 0.20.2+320 from cdh3 distro, right? Thank you. -Dmitriy On Tue, Sep 28, 2010 at 9:15 AM, Stack wrote: > On Mon, Sep 27, 2010 at 7:52 PM, Dmitriy Lyubimov > wrote: > > Hi, > > > > i

Re: WAL, failover questions

On Tue, Sep 28, 2010 at 9:15 AM, Stack wrote: > On Mon, Sep 27, 2010 at 7:52 PM, Dmitriy Lyubimov > wrote: > > Hi, > > > > i would be very grateful if somebody could clarify the following for me > > please. (0.20.5) > > > > yesterday we lost a short table (

WAL, failover questions

Hi, i would be very grateful if somebody could clarify the following for me please. (0.20.5) yesterday we lost a short table (~100 rows) in production without a trace. no matter how deep i looked in the logs of regionservers and the master, i haven't got a clue how it might have happened. When

Re: .META. not found

Ok, thanks, it fixed itself. Patience is the key i guess : ) thanks a bunch. but i see that a lot of people running into this problem with -ROOT- now and then. i wonder why that is. On Wed, Sep 22, 2010 at 3:10 PM, Dmitriy Lyubimov wrote: > Thanks. > > already tried this. >

Re: .META. not found

s cp ' to copy ROOT's hdfs directory from /hbase2 to > /hbase > * started HMaster back up > > apparently it doesn't care what the content of the ROOT region is on > startup > > > On Wed, Sep 22, 2010 at 5:45 PM, Dmitriy Lyubimov > wrote: > > > Hi, &

.META. not found

Hi, after restart of hbase cluster, which was running for a while (and perhaps under some stress), my hbase UI fails with the message below. the logs also have the same message. My other clients are spitting similar messages tried to do compactions, flush on .meta. table, which seem to succeed b

Re: Region servers exiting, not recovering

ices > we are dead, then splits our logs and reassigns the regions to other > nodes. This is the basis of how reliable hbase works in the face of > machine failure. > > -ryan > > On Tue, Sep 21, 2010 at 5:20 PM, Dmitriy Lyubimov > wrote: > > Hi, > > > > so i

Region servers exiting, not recovering

Hi, so in our production, we see temporary networking failures (we are not quite 100% sure what they are) but now and then region server's zookeeper session would get expired and in addition some ipc channels would throw 'channel closed'. This causes region server to exit. Which is not a very big