[
https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806472#action_12806472
]
Scott Carey commented on MAPREDUCE-1126:
----------------------------------------
b...@scott: the annotations for Input/OutputFormat seem to be misplaced. It
seems desirable to be able to write a single Map function that does wordcount
on Strings, regardless of whether those strings are stored in newline-delimited
text, sequence files, avro data files, or whatever.
Philip, yes, they are not in the right place. I just wanted to bring into the
conversation that 'SomeObject.setSomeBinding()' is not the only way to do these
sort of things. Annotations, unlike setter methods, can be moved around and
adapted to work in various ways without breaking APIs. For example, the
Input/OutputFormat annotation could go on either a Map class, OR some other
more specific annotation site, and with defaults and priority (set on
configuration > annotated on configuration > annotated on map > default)
determining which applies.
After thinking about it a bit more, and doing some research into how other APIs
do some tricky things with Annotations, there are a few things to consider.
* It is possible in some situations to infer the generic types of a class at
runtime by constructing an instance of an object with the same type arguments.
Example:
http://wiki.fasterxml.com/JacksonInFiveMinutes#Data_Binding_with_Generics.
* Annotations on class A can be applied to class B "Mix-In Annotations";
http://wiki.fasterxml.com/JacksonMixInAnnotations
* Post-compile time checks via an annotation processor can validate code before
run time in cases where the current M/R framework only breaks at run time.
What I think is most important to this discussion is that some layers of
configuration complexity can be hidden from users, and some of it deferred to
the future.
The 'site' of the configuration can be moved around with Annotations, opening
up ways to simplify the steps required to do declarative configuration.
With this in mind, some additional complexity to the procedural configuration
methods is more acceptable if there are good defaults and a later (backwards
compatible) API addition simplifies things. Likewise, some elements of
complexity can be skipped for now if it can be seen that those could be
available through a configuration extension later. Perhaps the procedural API
would never allow configuring a key and value to use different serializers to
avoid API complexity, but an annotation extension in the future allows that.
> shuffle should use serialization to get comparator
> --------------------------------------------------
>
> Key: MAPREDUCE-1126
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Reporter: Doug Cutting
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1126.2.patch, MAPREDUCE-1126.3.patch,
> MAPREDUCE-1126.4.patch, MAPREDUCE-1126.5.patch, MAPREDUCE-1126.6.patch,
> MAPREDUCE-1126.patch, MAPREDUCE-1126.patch
>
>
> Currently the key comparator is defined as a Java class. Instead we should
> use the Serialization API to create key comparators. This would permit,
> e.g., Avro-based comparators to be used, permitting efficient sorting of
> complex data types without having to write a RawComparator in Java.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
