Updated Ansible HBase installation scripts for Ubuntu 12.10

Hey there, I've updated the Ansible install/configure scripts for CDH4 HBase: It needs some cleanup over the next few days, but it should help anyone who's had to do a lot of workarounds to get the 12.04 scripts to work. https://github.com/LusciousPear/PalominoClusterTool -B --

Drawn to Scale HBaseCon Officewarming Party in SF

real-time SQL and search on HBase, so happy to help you with any HBase tech Qs you have. -- Bradford Stephens, CEO and Founder, Drawn to Scale http://drawntoscale.com (530) 763-DATA http://www.drawntoscale.com -- Spire: Real-Time Big Data

Re: "Error recovery for block... failed because recovery from primary datanode failed 6 times"

o show any reason here which is unusual. > > Anything in the master?  Did it time out this RS?  You're running with > replication = 1? > >> -Original Message- >> From: Bradford Stephens [mailto:bradfordsteph...@gmail.com] >> Sent: Sunday, February 13, 2011 10:

"Error recovery for block... failed because recovery from primary datanode failed 6 times"

toreFlusher.run(MemStoreFlusher.java:146) 2011-02-14 01:52:00,076 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished. 2011-02-14 01:52:00,139 WARN org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher: No longer connected to ZooKeeper, current state: Disconnected -- Br

Re: Slow MR data load to table

You have any monitoring of this >>> cluster going on?  Setting swappyness to zero from 60 is probably a >>> bit radical.  You want some swap if memory pressure.  60 is too loose. >>>  If you look at those killed map tasks... why they die?  Because >>> processes w

Re: Slow MR data load to table

toring of this > cluster going on?  Setting swappyness to zero from 60 is probably a > bit radical.  You want some swap if memory pressure.  60 is too loose. >  If you look at those killed map tasks... why they die?  Because > processes were killed by the kernel? > > St.Ack > >

Re: Slow MR data load to table

k prove their worth by hitting back. >  - Piet Hein (via Tom White) > > > > > -- Bradford Stephens, Founder, Drawn to Scale drawntoscalehq.com 727.697.7528 http://www.drawntoscalehq.com --  The intuitive, cloud-scale data solution. Process, store, query, search, and serve all yo

Re: Slow MR data load to table

l > machines are positively healthy and not swapping etc. - just to rule > out the (not so) obvious stuff. > > Lars > > On Mon, Dec 20, 2010 at 8:22 PM, Bradford Stephens > wrote: >> Aaaand, LZO is not enabled. >> >> On Mon, Dec 20, 2010 at 8:22 PM, Bradford

Re: Slow MR data load to table

Aaaand, LZO is not enabled. On Mon, Dec 20, 2010 at 8:22 PM, Bradford Stephens wrote: > FYI, here is the hbase-site: http://pastebin.com/z9aqy3dQ > > Also, in hbase-env: > > export HBASE_OPTS="-XX:+HeapDumpOnOutOfMemoryError > -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMo

Re: Slow MR data load to table

FYI, here is the hbase-site: http://pastebin.com/z9aqy3dQ Also, in hbase-env: export HBASE_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode" Hrm, that seems suboptimal On Mon, Dec 20, 2010 at 7:55 PM, Bradford Stephens wrote: > Greetings

Slow MR data load to table

a_set:ts, data_set:data, data_set:geo The code is simple (didn't write it): (Main): http://pastebin.com/vmPgeqNj (Mapper): http://pastebin.com/T2BQjs0k The logs are quite boring: HMaster: http://pastebin.com/zvyvNc3k Reigonserver: http://pastebin.com/QvJ4J7Ps Any ideas? -- Bradford Stephens,

Twitter Search + big Hadoop, Dec. 8th at Seattle Scalability Meetup

t 422 Yale Ave N. Address and more information here: http://www.meetup.com/Seattle-Hadoop-HBase-NoSQL-Meetup/ Look forward to seeing you all! Cheers, B -- Bradford Stephens, Founder, Drawn to Scale drawntoscalehq.com 727.697.7528 http://www.drawntoscalehq.com --  The intuitive, cloud-scale dat

Nodes up, Master sees 0 ReigonServers

es the proper 3 datanodes. Here's a Master log: http://pastebin.com/ZNPnmexF Here's a Reigonserver log: http://pastebin.com/PjUy4ra4 Any ideas? This was working properly with Hadoop .20.2. The new HDFS has been installed and formatted since then. Cheers, B -- Bradfor

Seattle Scalability Meetup: Rackspace OpenStack, Karmasphere Hadoop, Wed Oct 27

, Von Vorst Building, 426 Terry Ave N., Seattle, WA 98109-5210 Afterparty: Fierabend, 422 Yale Ave N -- Bradford Stephens, Founder, Drawn to Scale drawntoscalehq.com 727.697.7528 http://www.drawntoscalehq.com --  The intuitive, cloud-scale data solution. Process, store, query, search, and serve

Wed: Seattle Scalability / Hadoop / NoSQL Meetup: Killer Guests!

, there's delicious German beer and bratwurst at Fiererabend, 422 Yale Ave N., afterward. http://www.meetup.com/Seattle-Hadoop-HBase-NoSQL-Meetup/calendar/13704368/ Cheers, Bradford -- Bradford Stephens, Founder, Drawn to Scale drawntoscalehq.com 727.697.7528 http://www.drawntoscaleh

Re: Slow Inserts on EC2 Cluster

Ah, that explains a lot. Thanks for the tips JGray! I shall do that ASAP. On Thu, Sep 2, 2010 at 12:10 PM, Andrew Purtell wrote: >> From: Bradford Stephens >> A small improvement, but nowhere near what I'm used to, >> even from vague memories of old clusters on EC2.

Re: Slow Inserts on EC2 Cluster

les.  12 CF = 12 tables. > > -ryan > > On Wed, Sep 1, 2010 at 5:56 PM, Bradford Stephens > wrote: >> Good call JD!  We've gone from 20k inserts/minute to 200k. Much >> better! I still think it's slower than I'd want by about one OOM, but >> it's prog

Re: Slow Inserts on EC2 Cluster

the customer and see if they really have any sparse data that would benefit from its own ColumnFamily. Probably not. Cheers, B On Wed, Sep 1, 2010 at 5:37 PM, Bradford Stephens wrote: > Yeah, those families are all needed -- but I didn't realize the files > were so small. That's o

Re: Slow Inserts on EC2 Cluster

filesystem, on average 5MB, that together account for ~64MB which is > the default flush size (and then it generates tons of compactions > which makes it even worse). Do you really need all those families? Try > merging them and see the difference. > > J-D > > On Wed, Sep 1, 2010

Re: Slow Inserts on EC2 Cluster

best to avoid using Lucid on EC2 for now, then. > > FYI, the EC2 scripts that I use build AMIs based on Amazon's old FC8 AMI > (with updates). See http://github.com/apurtell/hbase-ec2 > >  - Andy > > > > > -- Bradford Stephens, Founder, Drawn to Scale

Re: Slow Inserts on EC2 Cluster

ve point of view.  Whether or not EC2 will last is uncertain, >> but cloud computing environments will definitely be around for a long >> time.  What would it take to make HBase resilient enough to take >> advantage of those environments?  Based on my experience and comments >> on

Re: Slow Inserts on EC2 Cluster

Wow, thanks. I didn't consider that ... I try to avoid the cloud if at all possible :) Cheers, B On Wed, Sep 1, 2010 at 4:14 AM, Andrew Purtell wrote: >> From: Bradford Stephens >> I'm banging my head against some perf issues on EC2. I'm >> using .20.6 on ASF

Slow Inserts on EC2 Cluster

these perf issues before. Ideas? I'm sure it's something painfully obvious to everyone but moi :) Here's some logs: NameNode: http://pastebin.com/j09CJQJJ DataNode: http://pastebin.com/XudWcaxW RS: http://pastebin.com/wXPBAjpu RS GC: http://pastebin.com/jqJyKAXq -- Bradford Step

JSONP and Stargate

Hey homies, I'm trying to write some JavaScript (which I know little about) to pull data out of HBase via Stargate via jQuery. To get around the "Single Origin Policy", I'm trying to do gets by using JSONP, which embeds/retrieves requests in

Seattle Hadoop Day! August 14th

for the community! (But please don't get a ticket unless you're sure you'll attend.) Hadoop Day is hosted by Drawn to Scale (http://drawntoscalehq.com), and sponsored by Amazon Web Services (http://aws.amazon.com), Miller-Perry, Cloudera (http://www.cloudera.com), and others. Chee