[
https://issues.apache.org/jira/browse/NIFI-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847220#comment-15847220
]
ASF GitHub Bot commented on NIFI-3420:
--------------------------------------
Github user ilganeli closed the pull request at:
https://github.com/apache/nifi/pull/1457
> NIFI Should support generating Hadoop-readable Lz4 outside of HDFS Write
> ------------------------------------------------------------------------
>
> Key: NIFI-3420
> URL: https://issues.apache.org/jira/browse/NIFI-3420
> Project: Apache NiFi
> Issue Type: New Feature
> Reporter: Ilya Ganelin
>
> Per https://issues.apache.org/jira/browse/HADOOP-12990 data stored in Lz4
> format on Hadoop is in a different format from the data generated by the Lz4
> CLI. The Lz4 CLI can also not be used to generate the Hadoop-compatible
> format.
> At the moment, NiFi does not support compression to Lz4 for streaming data.
> Although PutHdfs in the Hadoop processors supports writing out Lz4 to HDFS
> (assuming the appropriate codec exists), if data is instead being saved to
> something like S3 or simply streamed, there's no way to generate Lz4
> compressed data.
> If the Lz4 command line tool is used within a custom processor to perform Lz4
> conversion, this data will then not be readable on Hadoop if it's
> subsequently loaded to HDFS.
> A processor can be added that performs the conversion streaming data into the
> Lz4 format that IS readable on Hadoop by using the Hadoop Lz4 Codec to do the
> compression.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
