[
https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532243
]
Tom White commented on HADOOP-1986:
-----------------------------------
> Do you have a proposal for how to implement this?
If we follow Owen's suggestion then we can construct a map of types to
Serializer classes. Then, when running
MapTask or ReduceTask we can use the map to instantiate an appropriate
Serializer for each of the key and the value types.
> In particular, how can we handle something like Thrift, whose instances don't
> all implement some interface?
The target class would have to be Object. However, for this to work we would
need to have some notion of precedence so more specific subtypes (like
Writable) match first. Also, this wouldn't allow you to use two different
serialization frameworks whose instances only have a common type of Object. I'm
not sure how much of a problem this would be in practice though.
(I just had a look at a Thrift class, generated with release 20070917, and it
is tagged with java.io.Serializable. It would be more useful though it if
implemented an interface that defined the read/write fields.)
> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>
> Key: HADOOP-1986
> URL: https://issues.apache.org/jira/browse/HADOOP-1986
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Tom White
> Fix For: 0.16.0
>
>
> Currently Map Reduce programs have to use WritableComparable-Writable
> key-value pairs. While it's possible to write Writable wrappers for other
> serialization frameworks (such as Thrift), this is not very convenient: it
> would be nicer to be able to use arbitrary types directly, without explicit
> wrapping and unwrapping.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
