[
https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805143#action_12805143
]
Jay Booth commented on MAPREDUCE-1126:
--------------------------------------
For what it's worth, I really like setOutputKeyClass and while I understand the
distinction that a given type might map to more than one serialization method,
could we find a compromise that keeps the simplicity of a general
setOutputKeyClass while providing the flexibility that Doug's looking for?
This could have the added benefit of decoupling the class declaration from the
serialization declaration, since they're coupled but kind of separate.
Maybe something like:
// defaults to class-based introspection, if there are conflicts, Writable wins
for legacy reasons
job.setOutputKeyClass(Text.class)
job.setOutputValueClass(Text.class)
...
// explicitly use avro
job.setOutputKeyClass(AvroText.class)
job.setOutputValueClass(AvroText.class)
job.setOutputKeySerialization(new AvroSerializationSpec(conf,schema))
job.setOutputValueSerialization(new AvroSerializationSpec(conf,
Schema.create(Schema.Type.NULL))
Under this setup, the explicit Avro situation will be slightly wordier, but all
serialization formats could go in through a common object type
(SerializationSpec or something) and we wouldn't have to have a different
static setter for every different serialization type. It also ensures that
keyClass and valueClass are set the same way regardless of serialization type,
which I'd think is worth having if possible.
Another option would be to enforce the 4-liner version in all cases so that
people don't get unpredictable behavior based on class introspection, I could
see going either way on that one.. it's more robust but would break legacy
code and is more typing for a lot of common cases where there's no chance for
confusion.
For what it's worth, I cringe a little bit when doing Class.staticMethod(Job),
I'm always worried in the back of my mind about squashing already-resident
values. Yeah, I know, namespacing, but I generally feel like if something is
core functionality then it should be accounted for in the base Job class.
> shuffle should use serialization to get comparator
> --------------------------------------------------
>
> Key: MAPREDUCE-1126
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Reporter: Doug Cutting
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1126.2.patch, MAPREDUCE-1126.3.patch,
> MAPREDUCE-1126.4.patch, MAPREDUCE-1126.5.patch, MAPREDUCE-1126.6.patch,
> MAPREDUCE-1126.patch
>
>
> Currently the key comparator is defined as a Java class. Instead we should
> use the Serialization API to create key comparators. This would permit,
> e.g., Avro-based comparators to be used, permitting efficient sorting of
> complex data types without having to write a RawComparator in Java.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
