[
https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829222#action_12829222
]
Doug Cutting commented on MAPREDUCE-1126:
-----------------------------------------
Arun> allowing map-in-key and map-in-value to have different serialization
mechanisms seems like an over-kill. Do we have use-cases for such requirements?
I don't think this is a requirement. But neither am I convinced, long-term,
that separating keys and values is a requirement. But it is a legacy. We
already permit different class-specified serializations for keys and values.
This issue proposes to consistently alter how key and value serializations are
specified, to not be restricted to class-only, but rather to be consistent with
the serialization API. Doing this without permitting keys and values to have
different serializations would be harder than simply allowing it, as it would
require the introduction of a new concept, the serialization context, while the
current patch simply is the logical combination of the existing Mapreduce API
and the existing serialization API.
Perhaps one of the causes of this present tension is that Job.java is striving
to be two things at once: a high-level API for folks directly building Java
mapreduce applications and a low-level API for folks building higher-level
abstractions based on mapreduce. Perhaps we should instead add a new low-level
API. This might looks something like:
{code}
final class Split {
String[] locations;
ByteBuffer data;
}
interface MapEmitter { // implemented by kernel
emit(ByteBuffer datum, int partition);
}
interface CombineEmitter { // implemented by kernel
emit(ByteBuffer datum);
}
interface Job { // implemented by user code
Split[] getSplits();
void map(ByteBuffer splitData, MapEmitter out);
int compare(ByteBuffer x, ByteBuffer y);
boolean hasCombiner();
void combine(Iterator<Iterator<ByteBuffer>>, CombineEmitter)
void reduce(Iterator<Iterator<ByteBuffer>>, String attemptName);
void abandon(String[] attemptNames);
void commit(String[] attemptNames);
}
{code}
This is of course over-simplified. We'd need to pass a Context through to job
methods, so that they can report progress, etc. We'd also need to figure out
how the Job implementation itself is specified, serialized, reconstituted. Etc.
So perhaps a way around the current impasse is to devise a new, lower-level,
mapreduce API that will then support multiple higher-level APIs, including the
current API. Since there is resistance to evolving the existing API, it might
remain untouched as a legacy API. Folks who, e.g., wish to use Avro would use
a new higher-level MapReduce API that we'd develop on top of this shared
low-level runtime.
> shuffle should use serialization to get comparator
> --------------------------------------------------
>
> Key: MAPREDUCE-1126
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Reporter: Doug Cutting
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1126.2.patch, MAPREDUCE-1126.3.patch,
> MAPREDUCE-1126.4.patch, MAPREDUCE-1126.5.patch, MAPREDUCE-1126.6.patch,
> MAPREDUCE-1126.patch, MAPREDUCE-1126.patch
>
>
> Currently the key comparator is defined as a Java class. Instead we should
> use the Serialization API to create key comparators. This would permit,
> e.g., Avro-based comparators to be used, permitting efficient sorting of
> complex data types without having to write a RawComparator in Java.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
