First, thank you very much for the reply!
so, this is my input:
a\tb
b\tc
c\ta
In other words, a map function initially receives the whole string a\tb as
its value.
And it processes my input data correctly. I actually changed my reduce
function to simply emit merged pairs from a map's input for checking this.
However, when I tried to cross join cases where I have both to_'s and
from_'s (for example, a reducer gets the following pair <a, to_b ; from_c> )
by splitting each value provided by a reducer's iterator with split("_"), it
just didn't work. Even though without this additional logic reducer DOES
output these values <a, to_b ; from_c>, so it GETS them. The same split
thing works just fine for keys in a reduce function i.e. it discriminates
cases with a composite key like "a_b" from a simple key like "a." My guess
is that Hadoop should be sorting values for a reducer behind the scene and
this somehow messes up an initial character encoding. I'm using a Text class
as a serializable wrapper for my strings. I guess there is no other option
for it?)))
I wanna try to get rid of composite keys first (the last output.collect in a
map function) to make things a bit simpler and test it again then.
On Fri, Jul 16, 2010 at 9:16 AM, Jeff Bean <jwfb...@cloudera.com> wrote:
> Is the tab the delimiter between records or between keys and values on the
> input?
>
> in other words does the input file look like this:
>
> a\tb
> b\tc
> c\ta
>
> or does it look like this:
>
> a b\tb c\tc a\t
>
> ?
>
> Jeff
>
> On Thu, Jul 15, 2010 at 6:18 PM, Nikolay Korovaiko <korovai...@gmail.com
> >wrote:
>
> > Hi everyone,
> >
> > I hope this is the right place for my question. If not, please, feel free
> > to
> > ignore it ;) and I'm sorry for any inconvenience made :(
> >
> > I'm writing a simple program for enumerating triangles in directed graphs
> > for my project. First, for each input arc (e.g. a b, b c, c a, note: a
> tab
> > symbol serves as a delimiter) I want my map function output the following
> > pairs ([a, to_b], [b, from_a], [a_b, -1]):
> >
> > public void map(LongWritable key, Text value,
> >
> > OutputCollector<Text, Text> output,
> >
> > Reporter reporter) throws IOException {
> >
> > String line = value.toString();
> >
> > String [] tokens = line.split(" ");
> >
> > output.collect(new Text(tokens[0]), new Text("to_"+tokens[1]));
> >
> > output.collect(new Text(tokens[1]), new Text("from_"+tokens[0]));
> >
> > output.collect(new Text(tokens[0]+"_"+tokens[1]), new Text("-1"));
> >
> > }
> >
> > Now my reduce function is supposed to cross join all pairs that have both
> > to_'s and from_'s and to simply propogate any other pairs whose keys
> > contain
> > "_".
> >
> > public void reduce(Text key, Iterator<Text> values,
> >
> > OutputCollector<Text, Text> output,
> >
> > Reporter reporter) throws IOException {
> >
> > String key_s = key.toString();
> >
> > if (key_s.indexOf("_")>0)
> >
> > output.collect(key, new Text("completed"));
> >
> > else {
> >
> > HashMap <String, ArrayList<String>> lists = new HashMap
> > <String, ArrayList<String>> ();
> >
> > while (values.hasNext()) {
> >
> > String line = values.next().toString();
> >
> > String[] tokens = line.split("_");
> >
> > if (!lists.containsKey(tokens[0])) {
> >
> > lists.put(tokens[0], new ArrayList<String>());
> >
> > }
> > lists.get(tokens[0]).add(tokens[1]);
> >
> > }
> >
> > for (String t : lists.get("to"))
> >
> > for (String f : lists.get("from"))
> >
> > output.collect(new Text(t+"_"+f), key);
> >
> >
> > }
> >
> > }
> >
> > And this is where the most exciting stuff happens. tokens[1] yields an
> > ArrayOutOfBounds exception. If you scroll up, you can see that by this
> > point
> > the iterator should give values like "to_a", "from_b", "to_b", etc...
> when
> > I
> > just output these values, everything looks ok and I have "to_a",
> "from_b".
> > But split() don't work at all, moreover line.length() is always 1 and
> > indexOf("*") returns -1! The very same indexOf WORKS PERFECTLY for
> keys...
> > where we have pairs whose keys contain "_"* and look like "a_b", "b_c"
> >
> > I'm really puzzled with all this. MapReduce is supposed to save lives
> > making
> > everything simple. Instead I spent several hours to just spot this...
> >
> > I'd really appreciate your help, guys!!! Thanks in advance!
> >
>
