Thanks Mahesh for your help. Wondering if u can provide some insight with the below compare method using byte[] in the SecondarySort example:
public static class Comparator extends WritableComparator { public Comparator() { super(URICountKey.class); } public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) { return compareBytes(b1, s1, l1, b2, s2, l2); } } My question is in the below compare method that i have given we are comparing word1/word2 which makes sense but what about this byte[] comparison, is it right in assuming it converts each objects word1/word2/word3 to byte[] and compares them. If so is it for performance reason it is done. Could you please verify. Thanks Sai ________________________________ From: Mahesh Balija <balijamahesh....@gmail.com> To: user@hadoop.apache.org; Sai Sai <saigr...@yahoo.in> Sent: Saturday, 23 February 2013 5:23 AM Subject: Re: WordPairCount Mapreduce question. Please check the in-line answers... On Sat, Feb 23, 2013 at 6:22 PM, Sai Sai <saigr...@yahoo.in> wrote: > >Hello > > >I have a question about how Mapreduce sorting works internally with multiple >columns. > > >Below r my classes using 2 columns in an input file given below. > > > >1st question: About the method hashCode, we r adding a "31 + ", i am wondering >why is this required. what does 31 refer to. > This is how usually hashcode is calculated for any String instance (s[0]*31^(n-1) + s[1]*31^(n-2) + ... + s[n-1]) where n stands for length of the String. Since in your case you only have 2 chars then it will be a * 31^0 + b * 31^1. > >2nd question: what if my input file has 3 columns instead of 2 how would you >write a compare method and was wondering if anyone can map this to a real >world scenario it will be really helpful. > you will extend the same approach for the third column, public int compareTo(WordPairCountKey o) { int diff = word1.compareTo(o.word1); if (diff == 0) { diff = word2.compareTo(o.word2); if(diff==0){ diff = word3.compareTo(o.word3); } } return diff; } > > > > @Override > public int compareTo(WordPairCountKey o) { > int diff = word1.compareTo(o.word1); > if (diff == 0) { > diff = word2.compareTo(o.word2); > } > return diff; > } > > @Override > public int hashCode() { > return word1.hashCode() + 31 * word2.hashCode(); > } > > >****************************** > >Here is my input file wordpair.txt > >****************************** > >a b >a c >a b >a d >b d >e f >b d >e f >b d > >********************************** > > >Here is my WordPairObject: > >********************************* > >public class WordPairCountKey implements WritableComparable<WordPairCountKey> { > > private String word1; > private String word2; > > @Override > public int compareTo(WordPairCountKey o) { > int diff = word1.compareTo(o.word1); > if (diff == 0) { > diff = word2.compareTo(o.word2); > } > return diff; > } > > @Override > public int hashCode() { > return word1.hashCode() + 31 * word2.hashCode(); > } > > > public String getWord1() { > return word1; > } > > public void setWord1(String word1) { > this.word1 = word1; > } > > public String getWord2() { > return word2; > } > > public void setWord2(String word2) { > this.word2 = word2; > } > > @Override > public void readFields(DataInput in) throws IOException { > word1 = in.readUTF(); > word2 = in.readUTF(); > } > > @Override > public void write(DataOutput out) throws IOException { > out.writeUTF(word1); > out.writeUTF(word2); > } > > > @Override > public String toString() { > return "[word1=" + word1 + ", word2=" + word2 + "]"; > } > >} > >****************************** > >Any help will be really appreciated. >Thanks >Sai >