[
https://issues.apache.org/jira/browse/MAPREDUCE-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830205#action_12830205
]
Doug Cutting commented on MAPREDUCE-1462:
-----------------------------------------
> Serializations contain a serialize and deserialize method along with the
> Input/OutputStream to write the object to.
Where would these be stored? Are you proposing we add another file to each
job? Currently we have conf+splits+jar. Do we really want tasks to have to
open more files?
This proposes fundamental changes to job submission that should be addressed
elsewhere. We can achieve the goals of this issue using the existing
mechanisms, as in Tom & Aaron's patch to MAPREDUCE-1126. Changing job
submission in the way you suggest should be discussed separately, in
MAPREDUCE-1183.
> it doesn't lose all type safety of the string to string maps where a single
> misspelling can cause an attribute to not be found or an extra space can
> cause parse errors
We have long used string maps for Hadoop configuration. These strings are
generally written by programs that use constants to avoid misspellings. This
approach has long served HTTP, SMTP and unix enviroment variables as a
language-independent means of specifying parameters. If you want to change
this, it should be addressed systematically in MAPREDUCE-1183.
> the job would have the map outputs' class name configured as currently,
So we'd have a class name in the job configuration too? That seems redundant
and inconsistent with your misspelling concerns.
> the Job/JobContext would get a new setter/getter for getting the
> Serialization for each context.
HADOOP-6420 already provided a mechanism for this purpose. Tom & Aaron's patch
for HADOOP-1126 use this mechanism with a constant per context rather than an
enum. Would you prefer that an enum was used? Perhaps you could suggest that
there?
> Enable context-specific and stateful serializers in MapReduce
> -------------------------------------------------------------
>
> Key: MAPREDUCE-1462
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1462
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: task
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Attachments: h-1462.patch
>
>
> Although the current serializer framework is powerful, within the context of
> a job it is limited to picking a single serializer for a given class.
> Additionally, Avro generic serialization can make use of additional
> configuration/state such as the schema. (Most other serialization frameworks
> including Writable, Jute/Record IO, Thrift, Avro Specific, and Protocol
> Buffers only need the object's class name to deserialize the object.)
> With the goal of keeping the easy things easy and maintaining backwards
> compatibility, we should be able to allow applications to use context
> specific (eg. map output key) serializers in addition to the current type
> based ones that handle the majority of the cases. Furthermore, we should be
> able to support serializer specific configuration/metadata in a type safe
> manor without cluttering up the base API with a lot of new methods that will
> confuse new users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
