Hi,
I want to know how "MapTask.java" is implemented, especially
"MapOutputBuffer" class defined in "MapTask.java".
I've been trying to read "MapTask.java" after reading some references such
as "Hadoop definitive guide" and
"http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html";, but
it's quite tough to directly read the code without detailed comments.
As I know, when each intermediate (key, value) pair is generated by the
user-defined map function, the pair is written by "MapOutputBuffer" class
defined in "MapTask.java" with MapOutputBuffer.collect() invoked.
However, I can't understand what each variable defined in "MapOutputBuffer"
means.
What I've understood is as follows (* please correct any misunderstanding):
- The byte buffer "kvbuffer" is where each actual (partition, key, value)
triple is written.
- An integer array "kvindices" is called "accounting buffer", every three
elements of which save indices to the corresponding triple in "kvbuffer".
- Another integer array "kvoffsets" contains indices of triples in
"kvindices".
- "kvstart", "kvend", "kvindex" are used to point "kvindex"
- "bufstart", "bufend", "bufvoid", "bufindex", "bufmark" are used to point
"kvbuffer"
What I can't understand is the comments beside variable definitions.
= definitions of some variables
=
private volatile int kvstart = 0; // marks beginning of *spill*
private volatile int kvend = 0;// marks beginning of *collectable*
private int kvindex = 0; // marks end of *collected*
private final int[] kvoffsets; // indices into kvindices
private final int[] kvindices; // partition, k/v offsets into
kvbuffer
private volatile int bufstart = 0; // marks beginning of *spill*
private volatile int bufend = 0; // marks beginning of *collectable*
private volatile int bufvoid = 0; // marks the point where we should
stop
// reading at the end of the buffer
private int bufindex = 0; // marks end of *collected*
private int bufmark = 0; // marks end of *record*
private byte[] kvbuffer; // main output buffer
==
Q1)
What do the terms "spill", "collectable", and "collected" mean?
I guess, because map outputs continue to be written to the buffer while the
spill takes place, there must be at least two pointers: from where to write
map outputs and to where to spill data; but I don't know what those "spill"
"collectable", and "collected" mean exactly.
Q2)
Is it efficient to partition data first and then sort records inside each
partition?
Does it happen to avoid comparing expensive pair-wise key comparisons?
Q3)
Are there any documents containing explanations about how such internal
classes are implemented?
Thanks,
eastcirclek
