Re: reading a row with lots of cells (wide-table) causing out-of-memory error

2009-12-01 Thread Ryan Rawson
Your answers inline: On Tue, Dec 1, 2009 at 11:06 PM, Sujee Maniyam wrote: > Hi all, > > I have the following table >     user_id   => {  "ip_address",   "ref_url" } > column qualifiers are timestamps.  Created with default options > (BLOCKSIZE => '65536', ...etc) > > so a typical row looks like

reading a row with lots of cells (wide-table) causing out-of-memory error

2009-12-01 Thread Sujee Maniyam
Hi all, I have the following table user_id => { "ip_address", "ref_url" } column qualifiers are timestamps. Created with default options (BLOCKSIZE => '65536', ...etc) so a typical row looks like: 'user1' => { ip_address:t1 => value1

Re: hbase bulk writes

2009-12-01 Thread Stack
I do not know of schemas where there are more than 5 or 6 families. My guess is that there will be issues. One issue for sure is that we do not parallelize queries across families yet. The queries run in series so will be slow when lots of families involved. It shouldn't be hard to addres

Re: hbase bulk writes

2009-12-01 Thread Calvin
Thanks again for the responses. Stack: What's the issue with 25 families? I will mostly be accessing HBase as a map-reduce source and will be looking at ~5 column families at a time. Is there any documentation on column family limits in practice? -Calvin On Mon, Nov 30, 2009 at 7:38 PM, stack

Re: RE: HBase Installation Difficulties

2009-12-01 Thread Ryan Rawson
this looks like you have configured hbase to talk to a hadoop namenode that isn't there. Remove the hbase.rootdir entry and it will use a path in /tmp by default, not using hdfs. On Dec 1, 2009 12:53 PM, "Ratner, Alan S (IS)" wrote: Vaibhav, Thanks for the pointer. It looks like the Master s

RE: HBase Installation Difficulties

2009-12-01 Thread Ratner, Alan S (IS)
Vaibhav, Thanks for the pointer. It looks like the Master starts momentarily (jps can see it as well as HQuorumPeer) but then it crashes leaving just HQuorumPeer running. Some of the log file follows. If it matters, this is running within VirtualBox running Ubunto 8.04 with Windows XP as the

Re: regionserver disconnection

2009-12-01 Thread Zhenyu Zhong
The java vm version I am running on is still 1.6.0_11. We are scheduling an upgrade soon. The command we use to start the RegionServer is: /usr/local/jdk1.6.0_11/bin/java -Xmx4096m -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:+PrintTen

Re: regionserver disconnection

2009-12-01 Thread Jean-Daniel Cryans
The issue you saw is now HBASE-2022. J-D On Tue, Dec 1, 2009 at 7:50 AM, Zhenyu Zhong wrote: > So far I have been using gchisto to view the gc-log. > In my last RS disconnection, I saw a total GC about 457 seconds. But > individually, the max is 1340 ms, min is 0.527ms, avg is 48ms. > > The RS d

Re: regionserver disconnection

2009-12-01 Thread Patrick Hunt
Interesting, remind me, what is your current status of: java vm version? options you are providing to the JVM on startup (-XX -Xmx and the like - if you could provide the exact command line you use to start the jvm that would be nice to see) FYI: I've seen issues with use of incremental gc p

Re: regionserver disconnection

2009-12-01 Thread Zhenyu Zhong
So far I have been using gchisto to view the gc-log. In my last RS disconnection, I saw a total GC about 457 seconds. But individually, the max is 1340 ms, min is 0.527ms, avg is 48ms. The RS disconnection might be due to other reasons. I think J-D has been digging that. thanks zhenyu On Mon,