[
https://issues.apache.org/jira/browse/NIFI-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caleb Fenton updated NIFI-2637:
-------------------------------
Description:
Currently, when creating a Hadoop Sequence File, the compression codec is
hardcoded as DefaultCodec, which is deflate. It would be better if this was
configurable or the codec is inferred from the Hadoop configuration.
Here's the relevant line in code:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/SequenceFileWriterImpl.java#L100
Here's the relevant block of code where the writer is created with the
DefaultCodec:
{noformat}
final SequenceFile.Writer writer = SequenceFile.createWriter(configuration,
SequenceFile.Writer.stream(fsDataOutputStream),
SequenceFile.Writer.keyClass(Text.class),
SequenceFile.Writer.valueClass(InputStreamWritable.class),
SequenceFile.Writer.compression(compressionType, new DefaultCodec())))
{noformat}
Basically, I want to use Snappy for certain sequence files, and I can't do that
by changing Hadoop or processor configuration. It looks like I'll have to
change the source.
was:
Currently, when creating a Hadoop Sequence File, the compression codec is
hardcoded as DefaultCodec, which is deflate. It would be better if this was
configurable or the codec is inferred from the Hadoop configuration.
Here's the relevant line in code:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/SequenceFileWriterImpl.java#L100
Here's the relevant block of code where the writer is created with the
DefaultCodec:
{noformat}
final SequenceFile.Writer writer = SequenceFile.createWriter(configuration,
SequenceFile.Writer.stream(fsDataOutputStream),
SequenceFile.Writer.keyClass(Text.class),
SequenceFile.Writer.valueClass(InputStreamWritable.class),
SequenceFile.Writer.compression(compressionType, new DefaultCodec())))
{noformat}
> Allow CreateHadoopSequenceFile compression codec to be configurable
> -------------------------------------------------------------------
>
> Key: NIFI-2637
> URL: https://issues.apache.org/jira/browse/NIFI-2637
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Caleb Fenton
> Priority: Minor
>
> Currently, when creating a Hadoop Sequence File, the compression codec is
> hardcoded as DefaultCodec, which is deflate. It would be better if this was
> configurable or the codec is inferred from the Hadoop configuration.
> Here's the relevant line in code:
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/SequenceFileWriterImpl.java#L100
> Here's the relevant block of code where the writer is created with the
> DefaultCodec:
> {noformat}
> final SequenceFile.Writer writer = SequenceFile.createWriter(configuration,
>
> SequenceFile.Writer.stream(fsDataOutputStream),
> SequenceFile.Writer.keyClass(Text.class),
>
> SequenceFile.Writer.valueClass(InputStreamWritable.class),
>
> SequenceFile.Writer.compression(compressionType, new DefaultCodec())))
> {noformat}
> Basically, I want to use Snappy for certain sequence files, and I can't do
> that by changing Hadoop or processor configuration. It looks like I'll have
> to change the source.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
