RE: Speeding up Scans

Sorry for jumping in late, and perhaps out of context, but I'm pasting in some findings (reported to this list by us a while back) that helped us to get scans to perform very fast. Adjusting hbase.client.prefetch.limit was critical for us.: It's even more mysterious than w

RE: PENDING_CLOSE for too long

#x27;t accidently comment out code that sorts your region listing by key before you start merging.. the API will happily let you merge any two random regions.. creating lots of interesting overlaps :O Take care,   -stu ________ From: Geoff Hendrey To: user@

RE: PENDING_CLOSE for too long

code that sorts your region listing by key before you start merging.. the API will happily let you merge any two random regions.. creating lots of interesting overlaps :O Take care,   -stu ________ From: Geoff Hendrey To: user@hbase.apache.org Cc: user@hbase.apac

RE: PENDING_CLOSE for too long

It seems like the pending close messages can be ignored? And once I test my tool, and confirm I know a little bit about what I'm doing, maybe we could share notes? Take care,   -stu ________ From: Geoff Hendrey To: user@hbase.apache.org Cc: hbase-u...@hadoop.

tool to move out consecutive regions

- WrongRegionExceptions due to overlapping regions & holes in the regions. I just spent all day yesterday cribbing off of St.Ack's check_meta.rb script to write a java program to fix up overlaps & holes in an offline fashion (hbase down, directly on hdfs), and will start testing n

Re: PENDING_CLOSE for too long

t improvement in this > regard. > See HBASE-3368. > > Geoff: > Can you publish your tool on HBASE JIRA ? > > Thanks > > On Sat, Oct 29, 2011 at 2:35 PM, Geoff Hendrey wrote: > > > Sure. I posted the code many weeks back for a tool that will repair holes &g

Re: PENDING_CLOSE for too long

fingers!). > > It seems like the pending close messages can be ignored? > And once I test my tool, and confirm I know a little bit about what I'm > doing, maybe we could share notes? > > Take care, > -stu > > > > >

RE: scanner deadlock?

Stack Sent: Wednesday, September 14, 2011 8:47 AM To: user@hbase.apache.org Subject: Re: scanner deadlock? On Wed, Sep 14, 2011 at 8:42 AM, Geoff Hendrey wrote: > 17 MR nodes, 8 reducers per machine = 138 concurrent reducers. > (machines are 12-core, and I've found 8 reducers with 1GB allo

RE: scanner deadlock?

queue for the RPCs, it'll just reject calls > when its > handler count. > > Also send a listing of your hbase rootdir: hadoop fs -lsr /hbase > > St.Ack > > On Tue, Sep 13, 2011 at 11:40 PM, Geoff Hendrey wrote: >> As expected, J-D's suggestion basically caus

RE: scanner deadlock?

Subject: Re: scanner deadlock? Yeah. Ten handlers and no queue for the RPCs, it'll just reject calls when its > handler count. Also send a listing of your hbase rootdir: hadoop fs -lsr /hbase St.Ack On Tue, Sep 13, 2011 at 11:40 PM, Geoff Hendrey wrote: > As expected, J-D's sugg

RE: scanner deadlock?

spond(HBaseServer.java:792) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083) -Original Message- From: Geoff Hendrey [mailto:ghend...@decarta.com] Sent: Tuesday, September 13, 2011 10:56 PM To: user@hbase.apache.org Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod

RE: scanner deadlock?

13, 2011 at 10:25 PM, Geoff Hendrey wrote: > I've upgraded to HotSpot 64 Bit Server VM, with HBase 90.4 and all > recommended config changes (100 region server handlers, mslab enabled, etc). > No change, if anything it dies faster. Count of sockets in CLOSE_WAIT on > 50010 incr

RE: scanner deadlock?

.queue.size to 1 and hbase.regionserver.handler.count to a low number (let's say 10)? If payload is putting too much memory pressure, we'll know." ...though I'm not sure what I'm supposed to observe with these settings...but I'll try it and report on the outcome. -geoff -Origi

RE: scanner deadlock?

e.org/jira/browse/HDFS-1836 Best regards,    - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) >____ >From: Geoff Hendrey >To: user@hbase.apache.org >Cc: Tony Wang ; Rohit Nigam ; Parmod >Me

how to get encoded region ID

Hi - I am writing a tool (which I will share) to remove regions, and move their data off to the side in an HDFS backup area. We need this tool because we have a lot of regions which are garbage, and we just want to get rid of them en mass. However, I need to determine the string value of t

RE: scanner deadlock?

] On Behalf Of Stack Sent: Tuesday, September 13, 2011 9:32 AM To: user@hbase.apache.org Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd Subject: Re: scanner deadlock? On Tue, Sep 13, 2011 at 8:20 AM, Geoff Hendrey wrote: > ...but we don't have a slow region server. I'm asking

RE: scanner deadlock?

Agree: 1.6.0_u18 was totally broken. We definitely saw that too. -geoff -Original Message- From: Andrew Purtell [mailto:apurt...@apache.org] Sent: Monday, September 12, 2011 3:45 PM To: user@hbase.apache.org Subject: Re: scanner deadlock? > From: Sandy Pratt > TLDR: OpenJDK ~= Oracle

RE: scanner deadlock?

your ongoing help. -geoff -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: Monday, September 12, 2011 11:46 PM To: user@hbase.apache.org Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd Subject: Re: scanner deadlock? On Mon, Sep 12, 201

RE: scanner deadlock?

pache.org Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd Subject: Re: scanner deadlock? No slow datanode in your cluster? When stuff is slow, can you figure who all are trying to talk to? St.Ack On Mon, Sep 12, 2011 at 8:37 PM, Geoff Hendrey wrote: > OK Guys - > > We upgraded to

RE: scanner deadlock?

t one thing, can you set ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to a low number (let's say 10)? If payload is putting too much memory pressure, we'll know. J-D On Mon, Sep 12, 2011 at 8:37 PM, Geoff Hendrey wrote: > OK Guys - > > We upgraded to 90.4, an

RE: scanner deadlock?

OK Guys - We upgraded to 90.4, and made all the suggested config changes. The only thing we have not done yet, but will try soon, is switching from OpenJDK to the HotSpot JVM. Unfortunately, the problem recurs exactly as before. We will test with the HotSpot JVM shortly. -geoff -Original Mes

RE: scanner deadlock?

Hi - Will definitely switch to hotspot jre (as opposed to openjdk). WRT scanner: I made sure scanner caching was set to 1, but I also played with various levels of caching (up to 100). No impact on the problem. WRT Put: write buffer size is 10*1024*1024 (10 MB). Each row is about 750KB, so I expe

RE: scanner deadlock?

Hi - I thought that as long as I specified neither -client nor -server, that Server Class detection would automatically invoke the "-server" option. http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-clas s.html We are running 12-core AMD Opteron which is AMD64, so according to

RE: summary of issue/status

grepped my logs and there is no OOM. I also ran "lsof | wc -l" and response is 14000; we are nowhere near any limits ("ulimit -n" is 10), so ruled that out. -geoff From: Geoff Hendrey Sent: Sunday, September 11, 2011 5:52 PM To: 'hbase-u...@hadoop.apache.org

summary of issue/status

OK. Here is the summary of what I know: A region server, after some amount of scanning, can begin to get ClosedChannelException when it tries to respond to the client. Unfortunately, this only effects the response to the client. The region server apparently continues to tell zookeeper and say "

scanner deadlock?

Hi all - I'm still dealing with the saga of ScannerTimeoutException, UnknownScannerException, etc. I rewrote my code, in the hope that simply a different approach and some different code paths might yield better results. No change. I tried many variations (caching 1 row vs caching many rows, ch

RE: 2 rows in .META. with same startkey in hbase 90.3

In addition to "holes" in .META. we often see this problem too. Now that we've turned off splitting, we no longer have seen either duplicate rows nor holesI suppose that gives a clue as to the culprit: splitting...as far as my experience can show me. -geoff -Original Message- From: Jo

merge regions?

Hi - I think we're in a situation where we have too many regions (50,000+) on a 20 node cluster. Unfortunately I learned late that splitting should be disabled. Is there any way to "merge" existing regions to bring my region count down? -geoff

RE: ScannerTimeoutException

9/8/11 10:57 AM, "Geoff Hendrey" wrote: thanks: 1) I have splitting disabled 2) possible, but each reduce task uses its own JVM. Therefore, GC could explain an individual failure, but not the exponentially increasing failure rate. Further, I can concurrently start a second M/R

RE: ScannerTimeoutException

, September 08, 2011 10:06 AM To: user@hbase.apache.org; Geoff Hendrey Subject: Re: ScannerTimeoutException There are a few reasons why this occurs 1. Regionserver is very busy. Splits and compactions together suck precious CPU/IO cycles. 2. GC pauses. See if you can see any swaps

RE: ScannerTimeoutException

t: Re: ScannerTimeoutException How are you connecting to the table? Are you using the HTablePool? Best Regards, Sonal Crux: Reporting for HBase <https://github.com/sonalgoyal/crux> Nube Technologies <http://www.nubetech.co> <http://in.linkedin.com/in/sonalgoyal> On Thu, Sep 8, 2011 at 8:53 P

RE: ScannerTimeoutException

geoff -Original Message- From: Jan Lukavský [mailto:jan.lukav...@firma.seznam.cz] Sent: Thursday, September 08, 2011 8:34 AM To: Geoff Hendrey Cc: user@hbase.apache.org Subject: Re: ScannerTimeoutException Hi Geoff, I believe you don't need the scan.addColumn() when you add the who

RE: ScannerTimeoutException

s("V1")); scan.addColumn(Bytes.toBytes("V1"), Bytes.toBytes("cluster_map")); scan.addColumn(Bytes.toBytes("V1"), Bytes.toBytes("version_control_number")); -geoff -Original Message- From: Geoff Hendrey Sent: Tue

RE: What's .META.'s hole?

Deleted the closed region " + closedRegion.getRegionNameAsString()); admin.assign(newRegion.getRegionName(), true); //Assign the new region. System.out.println("Assigned the new region " + newRegion.getRegionNameAsString()); } catch (IOExceptio

Re: Site and Book updated

A little bladerunner action on the typography. I like it. Sent from my iPhone On Sep 7, 2011, at 4:10 PM, "Stack" wrote: > the logo don't look too bad, does it? > > > On Wed, Sep 7, 2011 at 3:29 PM, Doug Meil > wrote: > > > > Hi folks- > > > > Stack deployed the book update last night and t

Re: What's .META.'s hole?

2) where holes are (and overlaps and other kinds of meta problems). > > Can you point me to the jiras/code with fixup routines? > > Thanks, > Jon. > > > On Sun, Sep 4, 2011 at 8:15 PM, Geoff Hendrey wrote: > > > a "hole" means when you run "hbase

RE: ScannerTimeoutException

I'll try your suggestions! -Original Message- From: Jan Lukavský [mailto:jan.lukav...@firma.seznam.cz] Sent: Tuesday, September 06, 2011 9:48 AM To: user@hbase.apache.org Cc: Geoff Hendrey; hbase-u...@hadoop.apache.org Subject: Re: ScannerTimeoutException Hi Geoff, we are having

ScannerTimeoutException

Hi - I found some odd behavior with ResultScanner.next(). Usually the times for next() are couple hundred ms. But occasionally the call to next spikes VERY long. In fact, I have the timeout set to 60 seconds (6), but once in a while the call to next() itself is interrupted by the ScannerTim

RE: What's .META.'s hole?

a "hole" means when you run "hbase hbck" you see "chain of regions in table is broken; edges does not contain " What happens is the endrow of a entry in .META. points to a nonexistent start row. Based on StAck's steps for fixing the problem, we wrote tool to repair it. Essentially the tool fixes

RE: prevent region splits?

:46 PM To: user@hbase.apache.org Subject: Re: prevent region splits? On Sun, Sep 4, 2011 at 12:08 PM, Geoff Hendrey wrote: > great advice guys. appreciate it. Have made the changes to increase > storefile size. I'd also like to prevent rebalancing while I am running > my large M/R Put j

RE: prevent region splits?

ge. Then your regions >won't split. We use this method when copying 'live' hbase to make a >backup. > >-Jack > >On Sat, Sep 3, 2011 at 4:32 PM, Geoff Hendrey >wrote: >> Is there a way to prevent regions from splitting while we are running a >> mapreduce j

prevent region splits?

Is there a way to prevent regions from splitting while we are running a mapreduce job that does a lot of Puts? It seems that there is a lot of HDFS activity related to the splitting of regions while my M/R job is doing the puts. Is it sensible to disable splitting during the job that does lots of P

RE: PENDING_CLOSE for too long

t.timeoutmonitor.timeout" to 180 in hbase-site.xml. St.Ack On Fri, Sep 2, 2011 at 1:40 PM, Geoff Hendrey wrote: > In the master logs, I am seeing "regions in transition timed out" and > "region has been PENDING_CLOSE for too long, running forced unasign". > B

PENDING_CLOSE for too long

In the master logs, I am seeing "regions in transition timed out" and "region has been PENDING_CLOSE for too long, running forced unasign". Both of these log messages occur at INFO level, so I assume they are innocuous. Should I be concerned? -geoff

confirmed procedure for repairing hole in hbase metadata

the rows contain about 1 MB of data in 1 column. -geoff -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: Thursday, August 18, 2011 11:29 AM To: Rohit Nigam Cc: Geoff Hendrey; user@hbase.apache.org; Search Subject: Re: version mismatch

Re: version mismatch exception

This is awesome info!! Thank you!! Sent from my iPhone On Aug 18, 2011, at 11:29 AM, "Stack" wrote: > On Wed, Aug 17, 2011 at 1:12 PM, Rohit Nigam wrote: > > Hi St.Ack > > The region in the file System are good, all I am looking is to change the > > end key of that region in the .META. table

RE: version mismatch exception

Hi St.Ack, Keying off of what you said: " Did you update the info:regioninfo cell so it has a new hregioninfo with same start and end row? You know this makes a new region, rather than extend the range of the previous region? (So the old region will be in the filesystem still with the old data).

Re: "unlink" orpan row from .META.

. check_meta.rb > > St.Ack > > On Fri, Aug 12, 2011 at 6:08 PM, Stack wrote: > > On Fri, Aug 12, 2011 at 5:37 PM, Geoff Hendrey wrote: > >> Notice that endkey "X" doesn't exist anywhere as a startkey. I want to > >> fix this by doing a put that wil

Re: "unlink" orpan row from .META.

. Sent from my iPhone On Aug 12, 2011, at 6:08 PM, "Stack" wrote: > On Fri, Aug 12, 2011 at 5:37 PM, Geoff Hendrey wrote: >> Notice that endkey "X" doesn't exist anywhere as a startkey. I want to >> fix this by doing a put that will replace start/end pai

"unlink" orpan row from .META.

our table inconsistency is due to an orphaned row in .META. What do I mean startkey endkey === AB BC CX DE Notice that endkey "X" doesn't exist anywhere as a startkey. I want to fix this by doing a put that will replace start

RE: corrupt .logs block

. but at least it recovers all data and I can do full rowcount for the table. Jimmy. -- From: "Geoff Hendrey" Sent: Thursday, August 11, 2011 2:21 PM To: "Jinsong Hu" ; Subject: RE: corrupt .logs block > Hey - > > Our table behaves fine until we try to do a mapreduce j

RE: corrupt .logs block

;inconsistent" table, however, I am able to do a rowcount for the table and there is no problem. Jimmy ------ From: "Geoff Hendrey" Sent: Thursday, August 11, 2011 10:36 AM To: Cc: "Search" Subject: RE: corrupt .logs bloc

RE: corrupt .logs block

t the log could be carrying out into the filesystem into hfiles). St.Ack On Mon, Aug 8, 2011 at 4:20 PM, Geoff Hendrey wrote: > Ah. Thanks for that. No, I don't need the log anymore. I am aware of how > to flush a table from the hbase shell. But since the "fsck /" tells m

RE: corrupt .logs block

ables). -geoff -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: Monday, August 08, 2011 4:09 PM To: user@hbase.apache.org Subject: Re: corrupt .logs block On Sat, Aug 6, 2011 at 12:12 PM, Geoff Hendrey wrote: > I've got a corrupt

Re: corrupt .logs block

n(region_name.to_java_bytes, > java.lang.Boolean::valueOf(force)) >end > > Cheers > > On Sat, Aug 6, 2011 at 12:12 PM, Geoff Hendrey wrote: > >> Hey - >> >> >> >> I've got a corrupt HDFS block in a region server's ".logs" directory

corrupt .logs block

Hey - I've got a corrupt HDFS block in a region server's ".logs" directory. Also, "hbase hbck" reports 5 inconsistencies. .META. shows a region as offline, and it won't come online. What are my options. "hbck -fix" didn't do anything. It's not "live" data and I don't mind losing it. Howeve

RE: hbase + lucene?

BASE-3529 > > And let me chase Jason to post his slides. > > St.Ack > > On Tue, Jul 19, 2011 at 9:12 AM, Geoff Hendrey > wrote: > > Hi - > > > > > > > > At hadoop summit it was mentioned that there was a planning meeting for > > a project regar

hbase + lucene?

Hi - At hadoop summit it was mentioned that there was a planning meeting for a project regarding hbase and lucene. I believe the meeting was scheduled for the day after the summit. I wasn't able to attend, but I would like to keep abreast of what's going on in this regard. Anyone know anything

RE: distribution of regions to servers

f the hdfs balancer as opposed to the hbase region balancer that I was referring too. You could run a 0.90.3 master on your 0.90.1 cluster if you wantedthe master has the balancer code (but probably best to upgrade all to 0.90.3 anyways). St.Ack On Tue, Jun 7, 2011 at 3:02 PM, Geoff Hendrey wrote

RE: distribution of regions to servers

up a few times here in the past. The case has been well-made by now that the balancer needs to take into consideration the table a region is from and make an effort at distributing tables evenly across the cluster (There is an issue for this already). St.Ack On Tue, Jun 7, 2011 at 11:33 AM, Geoff

distribution of regions to servers

I have a table with a hundred or so regions. When I look in the hbase web ui, I see that all the regions are on one server. Of course we have many other tables and lots of data. Some tables seem to distribute their regions amongst many servers. I know there probably isn't a "pat" answer to thi

connection loss for /hbase

Hi Folks - I am experiencing "ConnectionLoss for /hbase" when I run a mapreduce job with around 100 mappers and TableInputFormat. I am running a fully distributed cluster, letting hbase manage zookeeper. I am running HBase 90.1. Is the solution, per this jira, https://issues.apache.org/jira/bro

RE: bulkloader zookeeper connectString

knows about my hbase" via the environment variables. -geoff -Original Message- From: Geoff Hendrey Sent: Wednesday, May 25, 2011 11:29 AM To: user@hbase.apache.org Subject: RE: bulkloader zookeeper connectString Thanks for the pointer. I read the doc, and somehow had miss

RE: bulkloader zookeeper connectString

etebulkload /path/to/hfileoutputformat-output tablename' You mentioned "It seems your hadoop doesn't know about hbase's configuration.". What's the remedy for that? -geoff -Original Message- From: Geoff Hendrey Sent: Wednesday, May 25, 2011 11:29 AM To: user@

RE: bulkloader zookeeper connectString

has the zookeeper configuration file if zookeeper is NOT managed by HBase). It seems your hadoop doesn't know about hbase's configuration. J-D On Tue, May 24, 2011 at 9:41 PM, Geoff Hendrey wrote: > Hi - > > > > How can I pass the zookeeper connectString to the complete

bulkloader zookeeper connectString

Hi - How can I pass the zookeeper connectString to the completebulkoad utility so that it will not try to use the default "localhost". -geof

RE: bulk loader question

- From: Geoff Hendrey [mailto:ghend...@decarta.com] Sent: Saturday, April 23, 2011 7:39 PM To: hbase-u...@hadoop.apache.org Subject: bulk loader question I am running the bulk loader of HBase 90.1 per: "$ hadoop jar hbase-VERSION.jar completebulkload /user/todd/myoutput mytable"

bulk loader question

I am running the bulk loader of HBase 90.1 per: "$ hadoop jar hbase-VERSION.jar completebulkload /user/todd/myoutput mytable" I see the expected "Trying to load HFile ..." for each of the bulkload HFiles. However, the table is inaccesible after the bulk laod process runs. For instance, in the

RE: hadoop.log.file

5:54 PM To: user@hbase.apache.org Subject: Re: hadoop.log.file The HBase log4j.properties doesn't have that, but it has hbase.log.file https://github.com/apache/hbase/blob/trunk/conf/log4j.properties Is it what you're talking about? Thx, J-D On Thu, Mar 31, 2011 at 5:48 PM, Geoff

RE: hadoop.log.file

#x27;re talking about? J-D On Thu, Mar 31, 2011 at 3:22 PM, Geoff Hendrey wrote: > Hi - > > > > I was wondering where I can find an explanation of what hbase logs to > hadoop.log.file. This file is defined in log4j.properties. I see > DFSClient logging to it, but I can'

hadoop.log.file

Hi - I was wondering where I can find an explanation of what hbase logs to hadoop.log.file. This file is defined in log4j.properties. I see DFSClient logging to it, but I can't locate a doc describing exactly what hadoop.log.file is for, by Hbase. -geoff

auto-restart regionservers

Is there a way in which regionservers can be made to automatically restart when they crash/shutdown. We see several shutdowns every day...just wondering if there is any mode in which the daemons can "autorestart". BTW, I'd be happy to pursue the *cause* of the crash/shutdown, but it just seems to b

RE: CDH3B4 "fixes" permissions

cial" to cloudera. Also you have better chances to get answers from cloudera people on their own lists :) Hope that helps, J-D On Thu, Mar 17, 2011 at 4:08 PM, Geoff Hendrey wrote: > Fair. I've seen discussion of CDH3 on this list which is why I pinged. > Is it bad form to dis

RE: CDH3B4 "fixes" permissions

e.org Subject: Re: CDH3B4 "fixes" permissions Good question, you might want to ask that to cloudera. J-D On Thu, Mar 17, 2011 at 4:00 PM, Geoff Hendrey wrote: > Hi - > > > > I just upgraded to CDH3B4. I noticed when I ran 'hadoop dfsadmin > -upgrade' that the l

CDH3B4 "fixes" permissions

Hi - I just upgraded to CDH3B4. I noticed when I ran 'hadoop dfsadmin -upgrade' that the logs on the datanodes showed that hadoop was "fixing" the permissions of my dfs storage disks to "rwx--". I am just wondering why it does this? I had been using a subdirectory of one of the disks for ha

RE: 0.90.1 hbase-default.xml

we'd been using the 20.3-append jar provided by Ryan Rawson. Just replaced the hadoop jar with this one http://people.apache.org/~rawson/repo/org/apache/hadoop/hadoop-core/0.20 .3-append-r964955-1240/ , in our hbase 0.89 lib directory, and had no problems. Can we continue doing that instead of usi

RE: 0.90.1 hbase-default.xml

;s a mess at the moment since the append branch doesn't have a release). You might also want to consider using CDH3b4, which has a compatible hadoop and hbase. J-D On Fri, Mar 4, 2011 at 3:10 PM, Geoff Hendrey wrote: > Any advise on this one? It occurs when I start HBase, then the master >

RE: 0.90.1 hbase-default.xml

27;t reuse them between versions (since it led to many problems). J-D On Fri, Mar 4, 2011 at 2:44 PM, Geoff Hendrey wrote: > Hi, > > > > I tried to use my hbase-default.xml from 0.89 with my new 0.90.1 > installation. I get a message stating "hbase-default.xml seems to be &g

RE: 0.90.1 hbase-default.xml

sorry -- just saw docs/upgrading.html. Please disregard my question. -geoff -Original Message- From: Geoff Hendrey [mailto:ghend...@decarta.com] Sent: Friday, March 04, 2011 2:45 PM To: hbase-u...@hadoop.apache.org Subject: 0.90.1 hbase-default.xml Hi, I tried to use my hbase

0.90.1 hbase-default.xml

Hi, I tried to use my hbase-default.xml from 0.89 with my new 0.90.1 installation. I get a message stating "hbase-default.xml seems to be from an old version of hbase(null), this version is 0.90.1. But 0.90.1 doesn't seem to have an hbase-default.xml file that it ships with (at least not in

RE: follow-up question on TotalOrderPartitioner

--Original Message----- From: Geoff Hendrey [mailto:ghend...@decarta.com] Sent: Wednesday, March 02, 2011 9:08 PM To: hbase-u...@hadoop.apache.org Subject: follow-up question on TotalOrderPartitioner I upgraded my client to 0.90.1 per the suggestion (although the server is still 0.89). I no lo

follow-up question on TotalOrderPartitioner

I upgraded my client to 0.90.1 per the suggestion (although the server is still 0.89). I no longer get a NullPointerException when I try to use TotalOrderPartitioner. However, I cannot get the TotalOrderPartitioner to actually create the partition file, even though a message is printed "had

RE: bug in backport? How to work around?

I tried 90.1 and it works (no NullPointerException). Thanks! -g -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: Tuesday, March 01, 2011 9:07 PM To: user@hbase.apache.org Cc: Geoff Hendrey Subject: Re: bug in backport? How to work around

RE: bug in backport? How to work around?

try comparing 0.90.20100924 to 0.90.1 to see if changes? Does your hbase have 'HBASE-3392 Update backport of InputSampler to reflect MAPREDUCE-1820'? If not, want to apply it -- or just fast forward your install to 0.90.1? St.Ack On Tue, Mar 1, 2011 at 8:38 PM, Geoff Hendrey wro

bug in backport? How to work around?

I am using HBase 0.89.20100924. I get a NullPointerException from org.apache.hadoop.hbase.mapreduce.hadoopbackport.InputSampler: [exec] Exception in thread "main" java.lang.NullPointerException [exec] at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyVa lue(

RE: getSplits question

er startup/shutdown/assignment times). So you > probably dont want 50,000 32MB regions. > > -ryan > > On Wed, Feb 9, 2011 at 11:46 PM, Geoff Hendrey wrote: > > Oh, I definitely don't *need* my own to run mapreduce. However, if I want to control the number of records handl

RE: getSplits question

e your own getSplits() method to run a map reduce, I never did at least... -ryan On Wed, Feb 9, 2011 at 11:36 PM, Geoff Hendrey wrote: > Are endrows inclusive or exclusive? The docs say exclusive, but then the > question arises as to how to form the last split for getSplits(). The > c

getSplits question

Are endrows inclusive or exclusive? The docs say exclusive, but then the question arises as to how to form the last split for getSplits(). The code below runs fine, but I believe it is omitting some rows, perhaps b/c of the exclusive end row. For the final split, should the endrow be null? I tried

RE: bulk loaded value

e.org/~stack/hbase-0.90.0-candidate-3/docs/apidocs/index.html > (Doc could be a bit better but you get the idea). > > By what you print below, it looks like its a zero-padded long and > we're outputting it in HEX with each byte of the long escaped. > > St.Ack > > On Mon

RE: Scalability problem with HBase

just curious what you mean by "reverse search index". -g -Original Message- From: Thibault Dory [mailto:dory.thiba...@gmail.com] Sent: Sunday, January 23, 2011 1:42 PM To: user@hbase.apache.org Subject: Scalability problem with HBase Hello, I'm currently testing the performances of HB

RE: IndexOuputFormat?

her, I'd just put a new jar in my lib. -g -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: Wednesday, January 19, 2011 3:13 PM To: user@hbase.apache.org Subject: Re: IndexOuputFormat? On Wed, Jan 19, 2011 at 3:00 PM, Geoff Hendrey w

RE: IndexOuputFormat?

browse/HBASE-2212 The current URL seems to be: https://github.com/akkumar/hbasene I don't know if this is still active, however. --gh On Wed, Jan 19, 2011 at 10:12 AM, Geoff Hendrey wrote: > Hi - > > > > I have downloaded 0.89 because I need to use > HFileOutputForma

IndexOuputFormat?

Hi - I have downloaded 0.89 because I need to use HFileOutputFormat.configureIncrementalLoad. I downloaded it from here: http://mirror.olnevhost.net/pub/apache//hbase/hbase-0.89.20100924/ However, I don't see these two cla

RE: bulk loaded value

ase-u...@hadoop.apache.org Subject: Re: bulk loaded value Any chance of a patch to the doc Geoff? If you ran into the issue, others will too. I'm glad you figured it. St.Ack On Tue, Jan 18, 2011 at 9:24 AM, Geoff Hendrey wrote: > Thanks for your response. Her is what happened. I didn&#

RE: bulk loaded value

on on emission: http://people.apache.org/~stack/hbase-0.90.0-candidate-3/docs/apidocs/index.html (Doc could be a bit better but you get the idea). By what you print below, it looks like its a zero-padded long and we're outputting it in HEX with each byte of the long escaped. St.Ack On Mon,

bulk loaded value

Hi - I am using 0.89 for bulk loading. In my reducer: ImmutableBytesWritable ibw = new ImmutableBytesWritable(key.copyBytes()); KeyValue kv = new KeyValue(key.copyBytes(), Bytes.toBytes(context.getConfiguration().get("fam", "count")), Bytes.toBytes("c"), 100L,"hello

RE: Cluster Wide Pauses

This is not an answer to your question, but just an anecdote on cluster pauses/slowdowns. We had horrible problems with cluster wide pauses. I think there were several keys to getting this resolved: 1) we used the default settings recommended for bulk inserts: http://people.apache.org/~jdcryans/HU

RE: HBase stability

We we're having no end to "buffet" of errors and stability problems with 20.3 when we ran big mapreduce jobs to insert data. Upgraded to 20.6 last week, and have not seen any instability. Just my anecdotal experience. -geoff -Original Message- From: Anze [mailto:anzen...@volja.net] Sent:

unable to disable WAL

Hi - I've noticed that even though my mapred job disabled the WAL, we still see HLog flushing. In my mapred job I do: setWriteToWal(false). However, I still see this is region server logs: "logFlusher took 51132ms optional sync'ing hlog" I'm observing a pattern of cascading failur

RE: Smallest production HBase cluster

Hi, I am running a 3 node cluster. HDFS datanode and Hbase regionserver are running on each node. The Hbase master and HDFS namenode run on different machines (not "different" in the sense of "not in the cluster". Just different in the sense of "not on the same box in the cluster").Quad core, 64-b

  1   2   >