[
https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805861#action_12805861
]
Chris Douglas commented on MAPREDUCE-1126:
------------------------------------------
bq. Why are proposals now focusing on allowing users to specify different
serialization factories?
Yes, you're right; I mixed up the naming.
{{s/SerializationFactory/SerializationBase/g}}. The setter formal with a vararg
is still attractive, {{setSerialization(ctxt, SerializationBase...)}}, so the
semantics are the same for configuring the {{SerializationFactory}} whether
there are one or many. Setting it with one is equivalent to setting the base
w/o a {{SerializationFactory}}, but it keeps the {{accept}} check. The purpose
is to configure a context-specific {{SerializationFactory}}; I'm liking Arun's
suggestion of limiting these hooks to record contexts, rather than all the user
types.
bq. [...] it's far more common to target the MapReduce execution engine with a
higher level language rather than the Java API. What you propose as the common
case, then, appears to be uncommon in practice.
As Arun points out, framework authors are uncommon, even if their users are
not. The possible performance improvements from open hooks in the framework are
compelling (if most are theoretical at the moment), but the user-facing API for
MapReduce shouldn't be the Job's serialization. A more abstract collector may
be an interesting extension/interface when the use cases are better understood.
----
@Tom
The much-simplified patch is a considerable improvement. Losing the
{{\*JobData}} classes effected a cleaner integration. Thanks for this.
> shuffle should use serialization to get comparator
> --------------------------------------------------
>
> Key: MAPREDUCE-1126
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Reporter: Doug Cutting
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1126.2.patch, MAPREDUCE-1126.3.patch,
> MAPREDUCE-1126.4.patch, MAPREDUCE-1126.5.patch, MAPREDUCE-1126.6.patch,
> MAPREDUCE-1126.patch, MAPREDUCE-1126.patch
>
>
> Currently the key comparator is defined as a Java class. Instead we should
> use the Serialization API to create key comparators. This would permit,
> e.g., Avro-based comparators to be used, permitting efficient sorting of
> complex data types without having to write a RawComparator in Java.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
