[
https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787799#action_12787799
]
Aaron Kimball commented on MAPREDUCE-1126:
------------------------------------------
To allow Avro-based comparators during the shuffle, a few changes need to take
place. Here's my survey of the scope of the problem and proposal for addressing
it.
HADOOP-6323 allows {{SerializationBase}} to provide {{RawComparator}}
instances. But for Avro to be germane here, the {{Serializer}} and
{{Deserializer}} instances used by {{MapOutputBuffer}} and other shuffle code
need to be updated as well. {{MapOutputBuffer}} and other places where the
{{Serializer}} and {{Deserializer}} interfaces are used should be changed to
use instead {{SerializerBase}} and {{DeserializerBase}}. This should be done by
changing the current calls to {{SerializationBase.getSerializer(keyClassName)}}
to use the newer {{getSerializer(metadata)}} method.
Currently the only metadata we store is the {{Serialization-Class}} that
identifies the intermediate key and value classes. This is implicitly set in,
e.g. {{Job.setMapOutputKeyClass()}}. (It actually just stores a class name, and
the {{getSerializer(className)}} method converts that to the appropriate
metadata map for {{getSerializer(metadata)}}.) HADOOP-6165 allows an arbitrary
map of metadata to be passed to the serialization framework. We should take
advantange of this map throughout the shuffling process. The job's
{{Configuration}} should be updated to embed a map a la the outline in
HADOOP-6165 (See Tom's comment, part 4(ii)) to include a "family" of
configuration parameters. Given the renaming of various parameters, I propose
to embed the metadata maps in {{mapreduce.map.output.key.metadata}} and
{{mapreduce.map.output.value.metadata}}.
The {{JobContext.setMapOutputKeyClass()}} method would instead of setting
{{mapreduce.map.output.key.class}}, set the appropriate metadata strings under
{{mapreduce.map.output.key.metadata}} or
{{mapreduce.map.output.value.metadata}} to use the {{WritableSerialization}}
implementation. If this metadata is not present, metadata could be inferred
from {{mapreduce.map.output.key/value.class}}, which should be marked as
deprecated.
{{JobContext}} would also include new methods which could be used instead of
{{setMapOutputKeyClass()}}, e.g. {{setMapOutputKeySchema()}} which would accept
an Avro schema and configure the shuffle to use AvroGenericSerialization for
this data. A generic {{setMapOutputKeySerializationMetadata()}} could be added
as well, which would accept any {{string -> string}} map of metadata for use
with other arbitrary {{SerializationBase}} implementations. (It might be used
internally by {{setMapOutputKeyClass()}} and {{setMapOutputKeySchema()}}.)
On the internal side, {{JobContext}} already provides a method
{{getSortComparator()}} which provides the appropriate {{RawComparator}}
implementation. But it does not directly provide {{SerializerBase}} or
{{DeserializerBase}} instances; it instead only returns the key and value class
names and clients of this (e.g., {{MapOutputBuffer}}) are expected to
instantiate the correct serialization objects themselves. I think that
{{JobContext}} should provide {{getMapOutputKeySerializer()}} and
{{getMapOutputKeyDeserializer()}} methods that correctly handle either the use
of {{mapreduce.map.output.key.metadata}} or {{mapreduce.map.output.key.class}}
to avoid duplication of this logic.
All {{SerializerBase}}, {{DeserializerBase}}, and {{RawComparator}}
implementations instantiated during shuffle would then be created via uses of
this new API.
Longer term, metadata maps are necessary for reducer output as well; e.g.,
{{mapreduce.reduce.output.key.metadata}}. This would allow full end-to-end Avro
use for user data. Similar methods for JobContext would need to be added to set
these parameters, and {{Job.setOutputKeyClass()}} et al. would need to be
updated to use this new API as well. But I think that this may be more
reasonably part of MAPREDUCE-815 than the current issue, which is concerned
with just the shuffle process.
> shuffle should use serialization to get comparator
> --------------------------------------------------
>
> Key: MAPREDUCE-1126
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Reporter: Doug Cutting
>
> Currently the key comparator is defined as a Java class. Instead we should
> use the Serialization API to create key comparators. This would permit,
> e.g., Avro-based comparators to be used, permitting efficient sorting of
> complex data types without having to write a RawComparator in Java.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
