Re: Error: It took too long to wait for the table

2010-11-24 Thread Hari Sreekumar
Hi Lars, Is the async version available in hbase-0.20.6 ASF version? It is still in development right? hari On Wed, Nov 24, 2010 at 4:09 PM, Lars George lars.geo...@gmail.com wrote: Hi Hari, Disabling a table simply takes time as all RSs need to report back that the regions are

Re: Error: It took too long to wait for the table

2010-11-24 Thread Hari Sreekumar
Hi Lars, Great! Thanks!! Is there any downside to increasing this value to too high? hari On Wed, Nov 24, 2010 at 5:15 PM, Lars George lars.geo...@gmail.com wrote: Oh, right, I keep forgetting who is on which version. This was added in HBASE-3112. So in that case, this is bound by

Re: Error: It took too long to wait for the table

2010-11-24 Thread Stack
If you up the config., the retry will be more tenacious. To be safer, you could make it so you have a client config. and one used by the servers. See conf/hbase-env.sh for HBASE_CONF_DIR for pointing at alternate conf directories. This disable/enable malarky will run better in 0.90.x St.Ack

Re: question about meta data query intensity

2010-11-24 Thread Andrew Purtell
The REST gateway (Stargate) is a long lived client. :-) It uses HTablePool internally so this will keep some warm table references around in addition to the region location caching that HConnectionManager does behind the scenes. (10 references, but this could be made configurable.) Best

Re: managing 5-10 servers

2010-11-24 Thread Jean-Daniel Cryans
Not just su.pr, but also stumbleupon.com which has the social layer. We do have memcached in front of HBase. Regarding blog posts about our setup, just search for stumbleupon hbase and you'll find tons. The most recent presentation that's available online is my talk at Hadoop World. Vid:

Re: question about meta data query intensity

2010-11-24 Thread Stack
If you are game for deploying an instrumented jar, we could log client lookups in .META. and try and figure if it profligate. St.Ack On Wed, Nov 24, 2010 at 9:25 AM, Jack Levin magn...@gmail.com wrote: Yes, but that does not alleviate CPU contention should there be too many queries to a single

Re: question about meta data query intensity

2010-11-24 Thread Jack Levin
Yes, I am game. -Jack On Wed, Nov 24, 2010 at 11:02 AM, Stack st...@duboce.net wrote: If you are game for deploying an instrumented jar, we could log client lookups in .META. and try and figure if it profligate. St.Ack On Wed, Nov 24, 2010 at 9:25 AM, Jack Levin magn...@gmail.com wrote:

Re: MR to load HBase running slowly in reduce

2010-11-24 Thread Jean-Daniel Cryans
I'm using http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/mapreduce/IdentityTableReducer.html Did you set it up with TableMapReduceUtil? Not explicitly set be me If you use TableMapReduceUtil, then it's set to 2MB by default, but looking at the RS logs the write buffer is

Re: REST compression support (was Re: question about meta data query intensity)

2010-11-24 Thread Jack Levin
Thats great! Thanks. this will help us reduce network context switching as we remove the need to pass a lot of uncompressed packets. -Jack On Wed, Nov 24, 2010 at 10:15 AM, Andrew Purtell apurt...@apache.org wrote: Regards compressing the HTTP transactions between the REST server and REST

Re: MR to load HBase running slowly in reduce

2010-11-24 Thread Tim Robertson
I'm using http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/mapreduce/IdentityTableReducer.html Did you set it up with TableMapReduceUtil? Not explicitly set be me If you use TableMapReduceUtil, then it's set to 2MB by default, but looking at the RS logs the write buffer

Re: Garbage collection issues

2010-11-24 Thread Todd Lipcon
On Wed, Nov 24, 2010 at 7:01 AM, Sean Sechrist ssechr...@gmail.com wrote: Hey guys, I just want to get an idea about how everyone avoids these long GC pauses that cause regionservers to die. What kind of java heap and garbage collection settings do you use? What do you do to make sure

Re: anyone have the biggest data size in production environment?

2010-11-24 Thread Jack Levin
We are at 35 TB now. -Jac On Wed, Nov 24, 2010 at 12:35 PM, Todd Lipcon t...@cloudera.com wrote: On Wed, Nov 24, 2010 at 9:48 AM, Hiller, Dean (Contractor) dean.hil...@broadridge.com wrote: I found some posts that were using hbase in production but I am really curious what data sizes

Re: Garbage collection issues

2010-11-24 Thread Sean Sechrist
Interesting. The settings we tried earlier today slowed jobs significantly, but no failures (yet). We're going to try the 512MB newSize and 60% CMSInitiatingOccupancyFraction. 1 second pauses here and there would be OK for us we just want to avoid the long pauses right now. We'll also do what

Re: REST compression support (was Re: question about meta data query intensity)

2010-11-24 Thread Andrew Purtell
Btw, does it mean, I can send in a compressed query? Or only receive compressed data from REST or both? Jetty's GzipFilter only wraps response handling. I tested to see if Jetty has some built in support for Content-Encoding: gzip for PUT or POST and it appears not: Error 415 Unsupported