Re: Read/Write Performance

2010-12-30 Thread Wayne
Lesson learned...restart thrift servers *after* restarting hadoop+hbase. On Thu, Dec 30, 2010 at 3:39 PM, Wayne wrote: > We have restarted with lzop compression, and now I am seeing some really > long and frequent stop the world pauses of the entire cluster. The load > requests for all regions a

Re: Read/Write Performance

2010-12-30 Thread Wayne
We have restarted with lzop compression, and now I am seeing some really long and frequent stop the world pauses of the entire cluster. The load requests for all regions all go to zero except for the meta table region. No data batches are getting in (no loads are occurring) and everything seems fro

Re: Java Commited Virtual Memory significally larged then Heap Memory

2010-12-30 Thread Andrey Stepachev
No, I'm not using LZO on this host. Only cloudera hadoop 0.20.2+320 + hbase 0.89.20100830 Digging google gives only hints, that jit or something in jvm can eat memory, but nothing concrete. pmap shows that some memory blocks are grow in size... but what are they, i can't imagine. 4010a000

Re: Read/Write Performance

2010-12-30 Thread Wayne
HBase Version 0.89.20100924, r1001068 w/ 8GB heap I plan to run for 1 week straight maxed out. I am worried about GC pauses, especially concurrent mode failures (does hbase/hadoop suffer these under extended load?). What should I be looking for in the gc log in terms of problem signs? The ParNews

Re: HBase Client error

2010-12-30 Thread Stack
What versions of hbase? The 'exception' is INFO level -- not WARN or ERROR -- coming up out of zookeeper. When does it happen? When you shut down your client? Close up of zk client connection can be a little awkward. This note from 0.90.0RC2 javadoc says a little on whats going on: http://peop

Re: Read/Write Performance

2010-12-30 Thread Stack
Oh, what versions are you using? St.Ack On Thu, Dec 30, 2010 at 9:00 AM, Stack wrote: > Keep going. Let it run longer.  Get the servers as loaded as you think > they'll be in production.  Make sure the perf numbers are not because > cluster is 'fresh'. > St.Ack > > On Thu, Dec 30, 2010 at 5:51 AM

Re: Read/Write Performance

2010-12-30 Thread Stack
Keep going. Let it run longer. Get the servers as loaded as you think they'll be in production. Make sure the perf numbers are not because cluster is 'fresh'. St.Ack On Thu, Dec 30, 2010 at 5:51 AM, Wayne wrote: > We finally got our cluster up and running and write performance looks very > good

Re: Java Commited Virtual Memory significally larged then Heap Memory

2010-12-30 Thread Todd Lipcon
Hi Andrey, Any chance you're using hadoop-lzo with CDH3b3? There was a leak in earlier versions of hadoop-lzo that showed up under CDH3b3. You should upgrade to the newest. If that's not it, let me know, will keep thinking. -Todd On Thu, Dec 30, 2010 at 12:13 AM, Andrey Stepachev wrote: > Add

Re: Read/Write Performance

2010-12-30 Thread Wayne
We are running 4 writers per region server node, 40 in total. We run them on the nodes themselves (calling a local thrift server) which avoids the network hops. On Thu, Dec 30, 2010 at 9:57 AM, Michael Russo wrote: > That's great to hear, thanks for sharing your results and configuration. > > H

Re: Read/Write Performance

2010-12-30 Thread Michael Russo
That's great to hear, thanks for sharing your results and configuration. How many concurrent writer processes are you running? Thanks, Michael On Thu, Dec 30, 2010 at 8:51 AM, Wayne wrote: > We finally got our cluster up and running and write performance looks very > good. We are getting susta

Re: Read/Write Performance

2010-12-30 Thread Wayne
We finally got our cluster up and running and write performance looks very good. We are getting sustained 8-10k writes/sec/node on a 10 node cluster from Python through thrift. These are values written to 3 CFs so actual hbase performance is 25-30k writes/sec/node. The nodes are currently disk i/o

Re: Realtime Query with hbase (aggregation)

2010-12-30 Thread Friso van Vollenhoven
If you need to do aggregates over substantial volumes of data, it will always take time. Scans for sequential rows are typically IO bound, so faster IO could probably make you go a bit faster, but on AWS I guess that's difficult. If you really want to speed things up, you need to find a way to r

Re: Java Commited Virtual Memory significally larged then Heap Memory

2010-12-30 Thread Andrey Stepachev
Addition information: ps shows, that my HBase process eats up to 4GB of RSS. $ ps --sort=-rss -eopid,rss | head | grep HMaster PID RSS 23476 3824892 2010/12/30 Andrey Stepachev > Hi All. > > After heavy load into hbase (single node, nondistributed test system) I got > 4Gb process size of

Java Commited Virtual Memory significally larged then Heap Memory

2010-12-30 Thread Andrey Stepachev
Hi All. After heavy load into hbase (single node, nondistributed test system) I got 4Gb process size of my HBase java process. On 6GB machine there was no room for anything else (disk cache and so on). Does anybody knows, what is going on, and how you solve this. What heap memory is set on you ho