[
https://issues.apache.org/jira/browse/MAPREDUCE-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805735#action_12805735
]
Scott Carey commented on MAPREDUCE-1126:
----------------------------------------
I am neck deep in building stuff on Avro. I've also got a custom Pig reader
that reads only my Avro record types as a prototyping stopgap until there is a
general solution.
Ted's Idea of a middle ground sounds useful. Special casing Writables is OK as
long as they don't have to be used. Making all the new stuff harder to use
sounds like a bad idea. Ideally, a schema system means as a user I never have
to write a Writable again. Those are annoying enough.
bq. For union types, it is a trivial restriction to limit map/reduce users to
wrap records around the unions
Wrappers are trivial to deal with in a schema declaration. They are not
trivial on the other side. First you have Foo and then Bar, and neither needs a
wrapper. But then they might because sometimes you want to serialize Bar in
columnar order and sometimes you don't. Now you have Bar, and ColulmnarBar.
Then you realize you need to have a union, so you make FooBarUnion. Then in
another use case you need BarFooUnion (different order, same Equals and
HashCode -- fun with permutations when the union is large).
Then you have composite types. FooBarStringNullUnion, and FooStringLongUnion.
Mapping serialization options to classes is not fun. Wrappers are a useful
design pattern for many purposes, but not for encapsulating one to many
relationships.
In general, users are moving away from writing directly to the MapReduce API
and using various frameworks. Making these frameworks high performance,
powerful, and expressive is more important IMO than preventing the low level
MapReduce API from getting a bit more complicated.
As for end-users writing MapReduce, the current situation is not all that
pretty anyway. 4 generic type arguments that must be perfectly aligned with
several type setting calls on a job configuration to avoid a runtime error?
The Map and Reduce classes have compile time checking on a few types, and thats
it.
Choosing a serialization is a declarative task, not a procedural one.
Annotations are what Java has right now for declarative metadata.
Unfortunately, very few people are experts at building annotation based
frameworks or using tools like ASM to enrich the capabilities. Has there been
any proposals to allow Annotations to handle this in a way cleaner than these
declarative setter methods?
I haven't thought through it that far, but here's a quick annotation based idea
that can bolt on to the work above or the current API. This is just a point
of contrast for this discussion, not a proposal to change -- perhaps it sparks
some ideas to simplify the user experience while also making the framework more
powerful. With the right tools those can go hand in hand.
WordCount with configurable Input/Output formats via annotations:
{code}
public class WordCount {
// if missing each of these has a default, or can be set with the traditional
setters
@Input(TextInputFormat.class);
@Output(AvroOutputFormat.class);
@InputWritables(key = LongWritable.class, val = Text.class).
@SchemaConfig(SchemaBasedJobData.class);
public static class Map extends MapReduceBase implements Mapper<LongWritable,
Text, Text, Integer> {
public void map(LongWritable key, Text value, OutputCollector<Text,
Integer> output, Reporter reporter) throws IOException {
. . .
}
}
}
@Input(AvroInputFormat.class);
@Output(AvroOutputFormat.class);
@SchemaConfig(SchemaBasedJobData.class);
public static class Reduce extends MapReduceBase implements Reducer<Text,
Integer, Text, Integer> {
public void reduce(Text key, Iterator<Integer> values,
OutputCollector<Text, Integer> output, Reporter reporter) throws IOException {
. . .
}
public static void main(String[] args) throws Exception {
JobConf conf = new JobConf(WordCount.class);
conf.setJobName("wordcount");
// infer inputs and outputs from annotations on the Map, Reduce, and
Combiner setters
// throw an error if these are not compatible.
conf.setMapperClass(Map.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);
// the "old way" can still work, setting individual input and output
classes. But it is not necessary when annotated and incompatible with schema
based systems.
FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));
JobClient.runJob(conf);
}
}
{code}
There are some things missing above.
For frameworks, configuration by method call isn't as big of a deal, but for
hand-written classes it is a good thing to keep the generic type declaration
and key/value class/type/schema declaration in the same place -- JobConf won't
tell you at compile time that you have screwed it up and misaligned types in
your Map. And generic types aren't available at runtime, increasing the number
of times the same information must be set.
Annotations can only set primitive types, Class, Enum, and String, as well as
arrays of these, so configuring a Schema becomes more difficult.
Schemas can only be stored in annotations if they are in String form -- which
may be too verbose to store in an annotation. I have one Schema that is over
4k in its Avro json form.
Anyhow, its just an idea for contrast. The debate is about how to deal with
declarative configuration in Java, so it seems relevant to bring up
Annotations.
> shuffle should use serialization to get comparator
> --------------------------------------------------
>
> Key: MAPREDUCE-1126
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1126
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Reporter: Doug Cutting
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1126.2.patch, MAPREDUCE-1126.3.patch,
> MAPREDUCE-1126.4.patch, MAPREDUCE-1126.5.patch, MAPREDUCE-1126.6.patch,
> MAPREDUCE-1126.patch, MAPREDUCE-1126.patch
>
>
> Currently the key comparator is defined as a Java class. Instead we should
> use the Serialization API to create key comparators. This would permit,
> e.g., Avro-based comparators to be used, permitting efficient sorting of
> complex data types without having to write a RawComparator in Java.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
