[
https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802564#action_12802564
]
Doug Cutting commented on MAPREDUCE-1126:
-----------------------------------------
> That is a really ugly change to the user's application.
It is moving something from kernel to library code, using the encouraged
pattern, to make the kernel data-representation agnostic. Avro applications
will be on equal footing with Writable applications. They'll do something like
AvroJobData.setMapInputSchema(job, mySchema). With generic data, the base
class isn't germane, but the schema is. Should we instead add that method
directly to Job, then do the same for Thrift, Protobuf, Hessian, Java
Serialization, etc?
This is analagous to the change from job.addInputPath() to
FileInputFormat.addInputPath().
These methods could be a bit more friendly, accepting the job directly rather
than requiring folks to call job.getConfiguration(). Perhaps Aaron can provide
a revised version that does this?
> We used to infer the serialization from the types. That was a clean model.
> Why are you trying to change it?
The model was changed a few months back, in HADOOP-6165. A class does not
uniquely identify a serialization. A class might implement Writable, but one
might wish to use Avro Reflection to serialize it. Avro also works for
built-in types like Long, null, etc. directly, as do other serialization
systems. Moreover, with union types (as, e.g., supported by Avro, Thrift and
perhaps others) there is no single class that corresponds to every input or
output datum.
There could, for example, be an Avro-based inputformat and outputformat that
represents all data types as byte[], containing the Avro binary data. Since
Avro comparators operate efficiently on binary data, this would perhaps be an
efficient way to use MapReduce as a sorting engine.
> Additionally, you have the framework depending on a library. That is a
> problem.
That's only for back-compatibility: what used to be the default is now in a
library. The only calls to library code are in deprecated Job setter methods,
invoked by user code. The framework never calls these setters.
If this is a problem, then we could duplicate the code from WritableJobData in
these deprecated methods. That's what was done when setInputPath() was
deprecated. Is that preferred?
> shuffle should use serialization to get comparator
> --------------------------------------------------
>
> Key: MAPREDUCE-1126
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Reporter: Doug Cutting
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1126.2.patch, MAPREDUCE-1126.3.patch,
> MAPREDUCE-1126.4.patch, MAPREDUCE-1126.5.patch, MAPREDUCE-1126.6.patch,
> MAPREDUCE-1126.patch
>
>
> Currently the key comparator is defined as a Java class. Instead we should
> use the Serialization API to create key comparators. This would permit,
> e.g., Avro-based comparators to be used, permitting efficient sorting of
> complex data types without having to write a RawComparator in Java.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
