[
https://issues.apache.org/jira/browse/HADOOP-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pete Wyckoff updated HADOOP-4065:
---------------------------------
Description:
Implement generic FlatFileDeserializationRecordReader which assumes a
Serialization Implementation is specific in the JobConf and that once
instantiated, that Serialization Implementation can figure out the actual
class being Deserialized from the JobConf. e.g., the JobConf specifies
RecordIOSerialization and then the specific class is LogRecordObject.
Another way one might to do this is to use the SerializationFactory to do the
lookup of the Serialization Implementation; however, this requires all
Serialization Implementations to be known apriori and registered and goes
against the spirit of a very generic FlatFileDeserializeRecordReader. (see
below re: adding Serialization implementations to contrib).
To ensure it is generic, I propose implementing the following Serialization
implementations:
1. RecordIOSerialization
2. LineReaderSerialization
3. ThriftSerialization
The first 2 should go in io/serialization and the Thrift one in contrib
somewhere.
was:
Implement generic FlatFileDeserializationRecordReader which assumes a
Serialization Implementation is specific in the JobConf and that once
instantiated, that Serialization Implementation can figure out the actual
class being Deserialized from the JobConf. e.g., the JobConf specifies
RecordIOSerialization and then the specific class is LogRecordObject.
Another way one might to do this is to use the SerializationFactory to do the
lookup of the Serialization Implementation; however, this requires all
Deserializers to be known apriori and registered and goes against the spirit of
a very generic FlatFileDeserializeRecordReader.
To ensure it is generic, I propose implementing the following Serialization
implementations:
1. RecordIOSerialization
2. LineReaderSerialization
3. ThriftSerialization
The first 2 should go in io/serialization and the Thrift one in contrib
somewhere.
> support for reading binary data from flat files
> -----------------------------------------------
>
> Key: HADOOP-4065
> URL: https://issues.apache.org/jira/browse/HADOOP-4065
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Joydeep Sen Sarma
> Attachments: HADOOP-4065.0.txt, HADOOP-4065.1.txt, ThriftFlatFile.java
>
>
> Implement generic FlatFileDeserializationRecordReader which assumes a
> Serialization Implementation is specific in the JobConf and that once
> instantiated, that Serialization Implementation can figure out the actual
> class being Deserialized from the JobConf. e.g., the JobConf specifies
> RecordIOSerialization and then the specific class is LogRecordObject.
> Another way one might to do this is to use the SerializationFactory to do the
> lookup of the Serialization Implementation; however, this requires all
> Serialization Implementations to be known apriori and registered and goes
> against the spirit of a very generic FlatFileDeserializeRecordReader. (see
> below re: adding Serialization implementations to contrib).
> To ensure it is generic, I propose implementing the following Serialization
> implementations:
> 1. RecordIOSerialization
> 2. LineReaderSerialization
> 3. ThriftSerialization
> The first 2 should go in io/serialization and the Thrift one in contrib
> somewhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
