>>Why do you need to know this? Were you trying to do a percentage of rows per region? Yes, that's exactly why I want to know this. Calculating % is the best way of distributing keys evenly across regions - me thinks. Everything else is just approximation.
>>Otherwise just have a member variable of your reducer class and increment it on each call to reduce(). Tried it, but didn't work. Basically, we should be able to find out in a reducer how many rows were created by all Mappers. I am a bit surprised MR framework doesn't provide this. >>I think you'll be better of finding a way to do it not using percentage if possible. Yes, if everything else fails, approximation is the way to go. >>Try calculating the size of the data instead perhaps. Correct, but as I said in previous email, I would rather not run another MR job just to calculate COUNT(*). There are over 8 Billion rows & sorting them is a bit expensive operation. >>we tallied up the KeyValue.getLenght() for each KeyValue in a row until the size reached a certain limit. The keyword in this line is "certain". How do you come up with that "certain" number, by approximation right? If everything else fails, that's what I will do. Thanks for your help. On Sun, May 13, 2012 at 9:35 AM, Bryan Beaudreault <bbeaudrea...@hubspot.com > wrote: > Why do you need to know this? Were you trying to do a percentage of rows > per region? Otherwise just have a member variable of your reducer class > and increment it on each call to reduce(). I think you'll be better of > finding a way to do it not using percentage if possible. Try calculating > the size of the data instead perhaps. You should have that available since > you are trying to bulkload anyway (which requires Put or KeyValue values, > both of which you can get the size from). > > On Sun, May 13, 2012 at 2:11 AM, Something Something < > mailinglist...@gmail.com> wrote: > > > Is there no way to find out inside a single reducer how many records were > > created by all the Mappers? I tried several ways but nothing works. For > > example, I tried this: > > > > reporter.getCounter(Task.Counter.REDUCE_INPUT_RECORDS).getValue(); > > > > It's not working for me. Should this have worked? Am I just doing > > something dumb? I would rather not create another MR job just to count # > > of lines. > > > > > > On Sat, May 12, 2012 at 7:07 PM, Bryan Beaudreault < > > bbeaudrea...@hubspot.com > > > wrote: > > > > > I did a very similar approach and it worked fine for me. Just spot > check > > > the regions after to make sure they look lexicographically sorted. I > > used > > > ImmutableBytesWritable as my key, and the default hadoop sorting for > that > > > turned out to sort lexicographically as required. Our hbase rows > varied > > in > > > size, so instead of doing a count of the number of rows, we tallied up > > the > > > KeyValue.getLenght() for each KeyValue in a row until the size reached > a > > > certain limit. > > > > > > On Sat, May 12, 2012 at 7:21 PM, Something Something < > > > mailinglist...@gmail.com> wrote: > > > > > > > Hello, > > > > > > > > This is really a MapReduce question, but the output from this will be > > > used > > > > to create regions for an HBase table. Here's what I want to do: > > > > > > > > Take an input file that contains data about users. > > > > Sort this file by a key (which consists of a few fields from the row) > > > > After every x # of rows write the key. > > > > > > > > > > > > Here's how I was going to structure my MapReduce: > > > > > > > > public Splitter { > > > > > > > > static int counter; > > > > > > > > private Mapper { > > > > map() { > > > > Build key by concatenating fields > > > > Write key > > > > increment counter; > > > > } > > > > } > > > > > > > > // # of reducers will be set to 1. My understanding is that this > > > will > > > > send the lines to reducer in sorted order one at a time - is this a > > > correct > > > > assumption? > > > > private Reducer { > > > > static long i; > > > > reduce() { > > > > static long splitSize = counter / 300; // 300 is region > > > size > > > > if (i == 0 || i == splitSize) { > > > > Write key; // this will be used as a 'startkey'. > > > > i = 0; > > > > } > > > > i++; > > > > } > > > > } > > > > } > > > > > > > > To summarize, there are 2 questions: > > > > > > > > 1) I am passing # of rows processed by Mapper to Reducer via a > static > > > > counter. Would this work? Is there a better way? > > > > 2) If I set # of reducers to 1, would the lines be sent to reducer > in > > > > sorted order one at a time? > > > > > > > > Thanks in advance for the help. > > > > > > > > > >