[
https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551237
]
eric baldeschwieler commented on HADOOP-1986:
---------------------------------------------
The code seems more readable on average after this work, so i think it
represents a step forward from where we were. Good stuff tom.
That said...
I just went through this whole thread. I find it quite mind bending. I remain
convinced that the use of templated types at all in our lowest level API is a
mistake and all of this templated foo should be an optional layer on top of a
basic byte oriented API. Other famous map-reduce implementations make that
choice (byte API only) and they get to bypass all of this discussion and
complexity. Something I ask everyone to think about. That is not a reason to
reject this patch, since I think this looks like an incremental improvement.
The code doesn't look like it adds any appreciable overhead, a function call
and a comparison per read seems likely to be in the noise, unless the stream
class bites us. Have you validated that we're not taking on extra cost here?
We're trying to strip out layers of copies and other framework fat.
> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>
> Key: HADOOP-1986
> URL: https://issues.apache.org/jira/browse/HADOOP-1986
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.16.0
>
> Attachments: hadoop-serializer-v2.tar.gz, SerializableWritable.java,
> serializer-v1.patch, serializer-v2.patch
>
>
> Currently Map Reduce programs have to use WritableComparable-Writable
> key-value pairs. While it's possible to write Writable wrappers for other
> serialization frameworks (such as Thrift), this is not very convenient: it
> would be nicer to be able to use arbitrary types directly, without explicit
> wrapping and unwrapping.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
