Re: anyway to turn off per-region metrics?
Thanks for the response Elliott, but I'm not sure how to use it. I tried adding the following: property namehbase.metrics.showTableName/name valuefalse/value /property to hbase-site.xml and then tried hbase.metrics.showTableName=false in hadoop-metrics.properties, but the metrics continue to show up. Could you please shed light on how to properly use your advice? Thanks, Oliver On 2013-07-30, at 12:20 AM, Elliott Clark ecl...@apache.org wrote: For 0.94.X you should use https://github.com/apache/hbase/blob/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java#L212 That will remove all of the per table metrics. For 0.95 and above this will be controlled by filters in the metrics properties file. On Mon, Jul 29, 2013 at 4:06 AM, Oliver Meyn (GBIF) om...@gbif.org wrote: Hi All, My ganglia server is being overwhelmed and I need to cut down on metrics. Is there a way to turn off the hbase.RegionServerDynamicStatistics.* metrics while keeping the hbase.regionserver.* metrics? I'm using cdh4.3, so hbase 0.94.6.1. Thanks, Oliver -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org
Re: Scan performance on compressed column families
Hi David, I wrote that blog post and I know that Lars George has much more experience than me with tuning HBase, especially in different environments, so weight our opinions accordingly. As he says, it will usually help, and the unusual cases of lower spec'd hardware (that I did those tests on) are where it might hurt scans, but obviously still helps with disk and network use. So take my post with a grain of salt, and as Kevin says, try it out on your data and your cluster and see what works best for you. Cheers, Oliver On 2012-11-03, at 3:57 PM, David Koch wrote: Hello, Are scans faster when compression is activated? The HBase book by Lars George seems to suggest so (p424, Section on Compression in chapter Performance Tuning). ... compression usually will yield overall better performance, because the overhead of the CPU performing the compression and de- compression is less than what is required to read more data from disk. I searched around for a bit and found this: http://gbif.blogspot.fr/2012/02/performance-evaluation-of-hbase.html. The author conducted a series of scan performance tests on tables of up to 200million rows and found that compression actually slowed down read performance slightly - albeit at lower CPU load. Thank you, /David -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org
Re: resource usage of ResultScanner's IteratorResult
On 2012-10-26, at 9:59 PM, Stack wrote:
On Thu, Oct 25, 2012 at 1:24 AM, Oliver Meyn (GBIF) om...@gbif.org wrote:
Hi all,
I'm on cdh3u3 (hbase 0.90.4) and I need to provide a bunch of row keys based
on a column value (e.g. give me all keys where column dataset = 1234).
That's straightforward using a scan and filter. The trick is that I want to
return an Iterator over my key type (Integer) rather than expose HBase
internals (i.e. Result), so I need some kind of Decorator that wraps the
IteratorResult. For every call to next() I'd then call the underlying
iterator's next() and extract my Integer key from the Result. That all
works fine, but what I'm wondering is what resources the IteratorResult is
holding, and how I can release those from my decorator.
In my current implementation the decorator's constructor looks like:
public OccurrenceKeyIterator(HTablePool tablePool, String
occurrenceTableName, Scan scan)
and the constructor builds the ResultScanner and subsequent iterator. In my
hasNext() method I can check the underlying iterator and if it says false I
can shutdown my scanner and return the table to the TablePool. But what if
the end-user never reaches the end of the Iterator, or just dereferences it?
Am I at risk of leaking tables, connections or anything else? Any tips on
what I should do?
If the close is not called, this is what will be missed on the HTable
instance:
flushCommits();
if (cleanupPoolOnClose) {
this.pool.shutdown();
}
if (cleanupConnectionOnClose) {
if (this.connection != null) {
this.connection.close();
}
}
this.closed = true;
In your case, the flushing of commits is of no import.
The pool above is an executor service inside of HTable used doing
batch calls. Again, you don't really use it but should probably get
cleaned up.
The connection close is good because though all HTables share a
Connection, the above close updates reference counters so we know when
we can let go of the connection.
Keep a list of what you've given out and if unused in N minutes, close
it yourself in background?
This kind of thing was all I could come up with but feels a bit messy. It
sounds like the only real consequence of not closing nicely is that the
reference counter doesn't get decremented, meaning the Connection wouldn't get
garbage collected if it were dereferenced. Is that right? That doesn't sound
too bad to me since the pool will be holding on to that connection anyway,
right? (Keeping in mind that the normal use case is everything gets cleaned
when end-user finishes iterating).
(when you fellas going to upgrade?)
It's definitely in the plan, but keeps getting pushed down in favour of getting
work done :) I read in the javadoc that the behaviour of tablepool and table
close changes in newer hbases - does my use case here change too (i.e. is it
even less dangerous to leave a table hanging in newer hbase)?
Thanks a lot for digging in to this Stack!
Oliver
--
Oliver Meyn
Software Developer
Global Biodiversity Information Facility (GBIF)
+45 35 32 15 12
http://www.gbif.org
resource usage of ResultScanner's IteratorResult
Hi all, I'm on cdh3u3 (hbase 0.90.4) and I need to provide a bunch of row keys based on a column value (e.g. give me all keys where column dataset = 1234). That's straightforward using a scan and filter. The trick is that I want to return an Iterator over my key type (Integer) rather than expose HBase internals (i.e. Result), so I need some kind of Decorator that wraps the IteratorResult. For every call to next() I'd then call the underlying iterator's next() and extract my Integer key from the Result. That all works fine, but what I'm wondering is what resources the IteratorResult is holding, and how I can release those from my decorator. In my current implementation the decorator's constructor looks like: public OccurrenceKeyIterator(HTablePool tablePool, String occurrenceTableName, Scan scan) and the constructor builds the ResultScanner and subsequent iterator. In my hasNext() method I can check the underlying iterator and if it says false I can shutdown my scanner and return the table to the TablePool. But what if the end-user never reaches the end of the Iterator, or just dereferences it? Am I at risk of leaking tables, connections or anything else? Any tips on what I should do? Thanks, Oliver -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org
Optimizing writes/compactions/storefiles
Hi all, We just spent some time figuring out how to get writes to work properly in our cluster on cdh3, and I wrote it up in a blog post. Might be of interest to some of you: http://gbif.blogspot.dk/2012/07/optimizing-writes-in-hbase.html Cheers, Oliver -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org
Re: Pre-split table using shell
Hi Simon,
I might be wrong but I'm pretty sure the splits file you specify is assumed to
be full of strings. So even though they look like bytes they're being
interpreted as the string value (like '\x00') instead of the actual byte \x00.
The only way I could get the byte representation of ints (in my case) to be
used for pre-splitting was to do it programatically.
Hope that helps,
Oliver
On 2012-06-12, at 11:41 AM, Simon Kelly wrote:
Yes, I'm aware that UUID's are designed to be unique and not evenly
distributed but I wouldn't expect a big gap in their distribution either.
The other thing that is really confusing me is that the regions splits
aren't lexicographical sorted. Perhaps there is a problem with the way I'm
specifying the splits in the split file. I haven't been able to find any
docs on what format the splits keys should be in so I've used what's
produced by Bytes.toStringBinary. Is that correct?
Simon
On 12 June 2012 10:23, Michael Segel michael_se...@hotmail.com wrote:
UUIDs are unique but not necessarily random and even in random samplings,
you may not see an even distribution except over time.
Sent from my iPhone
On Jun 12, 2012, at 3:18 AM, Simon Kelly simongdke...@gmail.com wrote:
Hi
I'm getting some unexpected results with a pre-split table where some of
the regions are not getting any data.
The table keys are UUID (generated using Java's UUID.randomUUID() ) which
I'm storing as a byte[16]:
key[0-7] = uuid most significant bits
key[8-15] = uuid least significant bits
The table is created via the shell as follows:
create 'table', {NAME = 'cf'}, {SPLITS_FILE = 'splits.txt'}
The splits.txt is generated using the code here:
http://pastebin.com/DAExXMDz which generates 32 regions split between
x00
and xFF. I have also tried with 16 byte regions keys (x00x00... to
xFFxFF...).
As far as I understand this should distribute the rows evenly across the
regions but I'm getting a bunch of regions with no rows. I'm also
confused
as the the ordering of the regions since it seems the start and end keys
aren't really matching up correctly. You can see the regions and the
requests they are getting here: http://pastebin.com/B4771g5X
Thanks in advance for the help.
Simon
--
Oliver Meyn
Software Developer
Global Biodiversity Information Facility (GBIF)
+45 35 32 15 12
http://www.gbif.org
Re: PerformanceEvaluation results
And here's a followup blog post with testing of our new cluster, and some more conclusions about PerformanceEvaluation: http://gbif.blogspot.dk/2012/06/faster-hbase-hardware-matters.html Cheers, Oliver On 2012-03-21, at 6:52 PM, Doug Meil wrote: Will do. On 3/20/12 12:55 PM, lars hofhansl lhofha...@yahoo.com wrote: We should like to this from the reference guide. - Original Message - From: Stack st...@duboce.net To: user@hbase.apache.org Cc: Sent: Tuesday, March 20, 2012 9:17 AM Subject: Re: PerformanceEvaluation results On Tue, Mar 20, 2012 at 8:53 AM, Oliver Meyn (GBIF) om...@gbif.org wrote: Apologies for responding to myself, but after some more testing I've concluded that we had a minor network bottleneck that was partially masking the real problem: not enough disks. Deductions based on ganglia metrics in a follow-up blog post: http://gbif.blogspot.com/2012/03/hbase-performance-evaluation-continued.h tml Nice post Oliver. St.Ack -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org
Re: HBase Performance Improvements?
Heya Something, I've put my ImportAvro class up for your amusement. It's a maven project so you should be able to check it out, build the jar with dependencies, and then just run it. See the Readme for more details. http://code.google.com/p/gbif-labs/source/browse/import-avro/ For your table splitting you're best off to create the table programatically using the starting keys you decided on using the loop you describe. Programatically is also the only way (I think) to pre-split a table with keys that aren't strings - eg byte representations of integers/longs. I wrote a little script to do that for our stuff too, which you can find in http://code.google.com/p/gbif-labs/source/browse/hbase-utils/trunk/src/main/java/org/gbif/hbase/util/TableCreator.java If you need help getting this stuff going please contact me directly rather than hitting the list. Cheers, Oliver On 2012-05-10, at 9:22 AM, Something Something wrote: Thank you Tim Bryan for the responses. Sorry for the delayed response. Got busy with other things. Bryan - I decided to focus on the region split problem first. The challenge here is to find the correct start key for each region, right? Here are the steps I could think of: 1) Sort the keys. 2) Count how many keys divide by # of regions we want to create. (e.g. 300). This gives us # of keys in a region (region size). 3) Loop thru the sorted keys every time region size is reached, write down region # starting key. This info can later be used to create the table. Honestly, I am not sure what you mean by hadoop does this automatically. If you used a single reducer, did you use secondary sort (setOutputValueGroupingComparator) to sort the keys? Did you loop thru the *values* to find regions? Would appreciate it if you would describe this MR job. Thanks. On Wed, May 9, 2012 at 8:25 AM, Bryan Beaudreault bbeaudrea...@hubspot.comwrote: I also recently had this problem, trying to index 6+ billion records into HBase. The job would take about 4 hours before it brought down the entire cluster, at only around 60% complete. After trying a bunch of things, we went to bulk loading. This is actually pretty easy, though the hardest part is that you need to have a table ready with the region splits you are going to use. Region splits aside, there are 2 steps: 1) Change your job to instead of executing yours Puts, just output them using context.write. Put is writable. (We used ImmutableBytesWritable as the Key, representing the rowKey) 2) Add another job that reads that input and configure it using HFileOutputFormat.configureIncrementalLoad(Job job, HTable table); This will add the right reducer. Once those two have run, you can finalize the process using the completebulkload tool documented at http://hbase.apache.org/bulk-loads.html For the region splits problem, we created another job which sorted all of the puts by the key (hadoop does this automatically) and had a single reducer. It stepped through all of the Puts calculating up the total size until it reached some threshold. When it did it recorded the bytearray and used that for the start of the next region. We used the result of this job to create a new table. There is probably a better way to do this but it takes like 20 minutes to write. This whole process took less than an hour, with the bulk load part only taking 15 minutes. Much better! On Wed, May 9, 2012 at 11:08 AM, Something Something mailinglist...@gmail.com wrote: Hey Oliver, Thanks a billion for the response -:) I will take any code you can provide even if it's a hack! I will even send you an Amazon gift card - not that you care or need it -:) Can you share some performance statistics? Thanks again. On Wed, May 9, 2012 at 8:02 AM, Oliver Meyn (GBIF) om...@gbif.org wrote: Heya Something, I had a similar task recently and by far the best way to go about this is with bulk loading after pre-splitting your target table. As you know ImportTsv doesn't understand Avro files so I hacked together my own ImportAvro class to create the Hfiles that I eventually moved into HBase with completebulkload. I haven't committed my class anywhere because it's a pretty ugly hack, but I'm happy to share it with you as a starting point. Doing billions of puts will just drive you crazy. Cheers, Oliver On 2012-05-09, at 4:51 PM, Something Something wrote: I ran the following MR job that reads AVRO files puts them on HBase. The files have tons of data (billions). We have a fairly decent size cluster. When I ran this MR job, it brought down HBase. When I commented out the Puts on HBase, the job completed in 45 seconds (yes that's seconds). Obviously, my HBase configuration is not ideal. I am using all the default HBase configurations that come out of Cloudera's distribution: 0.90.4+49. I am planning to read up on the following two: http
Re: HBase Performance Improvements?
Heya Something,
I had a similar task recently and by far the best way to go about this is with
bulk loading after pre-splitting your target table. As you know ImportTsv
doesn't understand Avro files so I hacked together my own ImportAvro class to
create the Hfiles that I eventually moved into HBase with completebulkload. I
haven't committed my class anywhere because it's a pretty ugly hack, but I'm
happy to share it with you as a starting point. Doing billions of puts will
just drive you crazy.
Cheers,
Oliver
On 2012-05-09, at 4:51 PM, Something Something wrote:
I ran the following MR job that reads AVRO files puts them on HBase. The
files have tons of data (billions). We have a fairly decent size cluster.
When I ran this MR job, it brought down HBase. When I commented out the
Puts on HBase, the job completed in 45 seconds (yes that's seconds).
Obviously, my HBase configuration is not ideal. I am using all the default
HBase configurations that come out of Cloudera's distribution: 0.90.4+49.
I am planning to read up on the following two:
http://hbase.apache.org/book/important_configurations.html
http://www.cloudera.com/blog/2011/04/hbase-dos-and-donts/
But can someone quickly take a look and recommend a list of priorities,
such as try this first...? That would be greatly appreciated. As
always, thanks for the time.
Here's the Mapper. (There's no reducer):
public class AvroProfileMapper extends AvroMapperGenericData.Record,
NullWritable {
private static final Logger logger =
LoggerFactory.getLogger(AvroProfileMapper.class);
final private String SEPARATOR = *;
private HTable table;
private String datasetDate;
private String tableName;
@Override
public void configure(JobConf jobConf) {
super.configure(jobConf);
datasetDate = jobConf.get(datasetDate);
tableName = jobConf.get(tableName);
// Open table for writing
try {
table = new HTable(jobConf, tableName);
table.setAutoFlush(false);
table.setWriteBufferSize(1024 * 1024 * 12);
} catch (IOException e) {
throw new RuntimeException(Failed table construction, e);
}
}
@Override
public void map(GenericData.Record record, AvroCollectorNullWritable
collector,
Reporter reporter) throws IOException {
String u1 = record.get(u1).toString();
GenericData.ArrayGenericData.Record fields =
(GenericData.ArrayGenericData.Record) record.get(bag);
for (GenericData.Record rec : fields) {
Integer s1 = (Integer) rec.get(s1);
Integer n1 = (Integer) rec.get(n1);
Integer c1 = (Integer) rec.get(c1);
Integer freq = (Integer) rec.get(freq);
if (freq == null) {
freq = 0;
}
String key = u1 + SEPARATOR + n1 + SEPARATOR + c1 + SEPARATOR +
s1;
Put put = new Put(Bytes.toBytes(key));
put.setWriteToWAL(false);
put.add(Bytes.toBytes(info), Bytes.toBytes(frequency),
Bytes.toBytes(freq.toString()));
try {
table.put(put);
} catch (IOException e) {
throw new RuntimeException(Error while writing to +
table + table., e);
}
}
logger.error( Finished processing user: + u1);
}
@Override
public void close() throws IOException {
table.close();
}
}
--
Oliver Meyn
Software Developer
Global Biodiversity Information Facility (GBIF)
+45 35 32 15 12
http://www.gbif.org
Re: Doumentation broken
Looks like /book got moved under another /book, so something is definitely wrong. You can try an unstyled version at: http://hbase.apache.org/book/book/book.html Cheers, Oliver On 2012-04-13, at 9:59 AM, Nitin Pawar wrote: Hello, Is there any maintenance going on with hbase.apache.org? All the links (ex: http://hbase.apache.org/book/architecture.html#arch.overview) are return 404 NOT FOUND Thanks, Nitin Pawar
Re: PerformanceEvaluation results
Apologies for responding to myself, but after some more testing I've concluded that we had a minor network bottleneck that was partially masking the real problem: not enough disks. Deductions based on ganglia metrics in a follow-up blog post: http://gbif.blogspot.com/2012/03/hbase-performance-evaluation-continued.html Cheers, Oliver On 2012-02-28, at 5:10 PM, Oliver Meyn (GBIF) wrote: Hi all, I've spent the last couple of weeks working with PerformanceEvaluation, trying to understand scan performance in our little cluster. I've written a blog post with the results and would really welcome any input you may have. http://gbif.blogspot.com/2012/02/performance-evaluation-of-hbase.html Cheers, Oliver -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org
ethernet channel bonding experiences
Hi all, I've been experimenting with PerformanceEvaluation in the last weeks and on a whim thought I'd give channel bonding a try to see if it was networking bandwidth that was acting as the bottleneck. It would seem that it's not quite as trivial as it sounds, so I'm looking for other people's experience in using HBase over bonded network channels. Specifically I'd be keen to hear what mode you used [1], given that the Dell/Cloudera document [2] suggests mode 6, and especially if you've resorted to mode 4 (802.3ad - hardware). My initial experiments with mode 6 (after a bunch of flailing around in the dark) suggest only nominal improvements when using PE in pure scan tests (5-10%, which is probably within test error). Thanks, Oliver [1] http://www.kernel.org/doc/Documentation/networking/bonding.txt [2] PDF: http://dell.cloudera.com/wp-content/uploads/2011/09/Dell_Cloudera_solution_for_Apache_Hadoop_Reference_Architecture_v1.2.pdf -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org
PerformanceEvaluation results
Hi all, I've spent the last couple of weeks working with PerformanceEvaluation, trying to understand scan performance in our little cluster. I've written a blog post with the results and would really welcome any input you may have. http://gbif.blogspot.com/2012/02/performance-evaluation-of-hbase.html Cheers, Oliver -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org
Re: strange PerformanceEvaluation behaviour
On 2012-02-15, at 5:39 PM, Stack wrote: On Wed, Feb 15, 2012 at 1:53 AM, Oliver Meyn (GBIF) om...@gbif.org wrote: So hacking around reveals that key collision is indeed the problem. I thought the modulo part of the getRandomRow method was suspect but while removing it improved the behaviour (I got ~8M rows instead of ~6.6M) it didn't fix it completely. Since that's really what UUIDs are for I gave that a shot (i.e UUID.randomUUID()) and sure enough now I get the full 10M rows. Those are 16-byte keys now though, instead of the 10-byte that the integers produced. But because we're testing scan performance I think using a sequentially written table would probably be cheating and so will stick with randomWrite with slightly bigger keys. That means it's a little harder to compare to the results that other people get, but at least I know my internal tests are apples to apples. Oh and I removed the outer 10x loop and that produced the desired number of mappers (ie what I passed in on the commandline) but made no difference in the key generation/collision story. Should I file bugs for these 2 issues? Thanks Oliver for digging. Using UUIDs will make it tougher on the other end when reading? How do you divide up the UUID space? UUIDs are not well distributed across the possible key space IIUC. Should writing UUIDs be an option on PE? Thanks again for figuring it. St.Ack Honestly I don't know very much about UUIDs so I didn't consider their distribution over the keyspace - just used UUID.randomUUID() and more or less crossed my fingers :) I agree that they're a bit of a PITA when it comes to reading them, but I think having exactly the expected number of rows written and read in the test makes the PE more obvious and therefore more useful/useable. So yes, I think UUIDs as a key option in PE would be good. I left some code in the JIRA as a starting point for a patch. Cheers, Oliver
Re: strange PerformanceEvaluation behaviour
On 2012-02-15, at 7:32 AM, Stack wrote: On Tue, Feb 14, 2012 at 8:14 AM, Stack st...@duboce.net wrote: 2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B). I just tried it here and got similar result. I wonder if its the randomWrite? What if you do sequentialWrite, do you get our 10M? Thanks for checking into this stack - when using sequentialWrite I get the expected 10485700 rows. I'll hack around a bit on the PE to count the number of collisions, and try to think of a reasonable solution. Thanks, Oliver
Re: strange PerformanceEvaluation behaviour
On 2012-02-15, at 9:09 AM, Oliver Meyn (GBIF) wrote: On 2012-02-15, at 7:32 AM, Stack wrote: On Tue, Feb 14, 2012 at 8:14 AM, Stack st...@duboce.net wrote: 2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B). I just tried it here and got similar result. I wonder if its the randomWrite? What if you do sequentialWrite, do you get our 10M? Thanks for checking into this stack - when using sequentialWrite I get the expected 10485700 rows. I'll hack around a bit on the PE to count the number of collisions, and try to think of a reasonable solution. So hacking around reveals that key collision is indeed the problem. I thought the modulo part of the getRandomRow method was suspect but while removing it improved the behaviour (I got ~8M rows instead of ~6.6M) it didn't fix it completely. Since that's really what UUIDs are for I gave that a shot (i.e UUID.randomUUID()) and sure enough now I get the full 10M rows. Those are 16-byte keys now though, instead of the 10-byte that the integers produced. But because we're testing scan performance I think using a sequentially written table would probably be cheating and so will stick with randomWrite with slightly bigger keys. That means it's a little harder to compare to the results that other people get, but at least I know my internal tests are apples to apples. Oh and I removed the outer 10x loop and that produced the desired number of mappers (ie what I passed in on the commandline) but made no difference in the key generation/collision story. Should I file bugs for these 2 issues? Thanks, Oliver
Re: strange PerformanceEvaluation behaviour
Okie: 10x # of mappers: https://issues.apache.org/jira/browse/HBASE-5401 wrong row count: https://issues.apache.org/jira/browse/HBASE-5402 Oliver On 2012-02-15, at 11:50 AM, yuzhih...@gmail.com wrote: Oliver: Thanks for digging. Please file Jira's for these issues. On Feb 15, 2012, at 1:53 AM, Oliver Meyn (GBIF) om...@gbif.org wrote: On 2012-02-15, at 9:09 AM, Oliver Meyn (GBIF) wrote: On 2012-02-15, at 7:32 AM, Stack wrote: On Tue, Feb 14, 2012 at 8:14 AM, Stack st...@duboce.net wrote: 2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B). I just tried it here and got similar result. I wonder if its the randomWrite? What if you do sequentialWrite, do you get our 10M? Thanks for checking into this stack - when using sequentialWrite I get the expected 10485700 rows. I'll hack around a bit on the PE to count the number of collisions, and try to think of a reasonable solution. So hacking around reveals that key collision is indeed the problem. I thought the modulo part of the getRandomRow method was suspect but while removing it improved the behaviour (I got ~8M rows instead of ~6.6M) it didn't fix it completely. Since that's really what UUIDs are for I gave that a shot (i.e UUID.randomUUID()) and sure enough now I get the full 10M rows. Those are 16-byte keys now though, instead of the 10-byte that the integers produced. But because we're testing scan performance I think using a sequentially written table would probably be cheating and so will stick with randomWrite with slightly bigger keys. That means it's a little harder to compare to the results that other people get, but at least I know my internal tests are apples to apples. Oh and I removed the outer 10x loop and that produced the desired number of mappers (ie what I passed in on the commandline) but made no difference in the key generation/collision story. Should I file bugs for these 2 issues? Thanks, Oliver
strange PerformanceEvaluation behaviour
Hi all, I've been trying to run a battery of tests to really understand our cluster's performance, and I'm employing PerformanceEvaluation to do that (picking up where Tim Robertson left off, elsewhere on the list). I'm seeing two strange things that I hope someone can help with: 1) With a command line like 'hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 10' I see 100 mappers spawned, rather than the expected 10. I expect 10 because that's what the usage text implies, and what the javadoc explicitly states - quoting from doMapReduce Run as many maps as asked-for clients. The culprit appears to be the outer loop in writeInputFile which sets up 10 splits for every asked-for client - at least, if I'm reading it right. Is this somehow expected, or is that code leftover from some previous iteration/experiment? 2) With that same randomWrite command line above, I would expect a resulting table with 10 * (1024 * 1024) rows (so 10485700 = roughly 10M rows). Instead what I'm seeing is that the randomWrite job reports writing that many rows (exactly) but running rowcounter against the table reveals only 6549899 rows. A second attempt to build the table produces slightly different results (e.g. 6627689). I see a similar discrepancy when using 50 instead of 10 clients (~35% smaller than expected). Key collision could explain it, but it seems pretty unlikely (given I only need e.g. 10M keys from a potential 2B). Any and all input appreciated. Thanks, Oliver -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org
Re: snappy error during completebulkload
Thanks Todd, that makes more sense now. I gave up on trying to build the native libraries on os x (not officially supported presumably because it's such a PITA) and instead ran from a centos machine and that worked flawlessly out of the box. Cheers, Oliver On 2012-01-09, at 7:21 PM, Todd Lipcon wrote: On Mon, Jan 9, 2012 at 2:42 AM, Oliver Meyn (GBIF) om...@gbif.org wrote: It seems really weird that compression (native compression even moreso) should be required by a command that is in theory moving files from one place on a remote filesystem to another. Any light shed would be appreciated. The issue is that the completebulkload script does actually open the files to read their metadata as well as the first/last key in the file. This is necessary to figure out which region each file belongs in. So, you do need the compression support on whatever machine you run completebulkload from. -Todd -- Todd Lipcon Software Engineer, Cloudera -- Oliver Meyn Software Developer Global Biodiversity Information Facility (GBIF) +45 35 32 15 12 http://www.gbif.org
