[jira] (NIFI-3420) NIFI Should support generating Hadoop-readable Lz4 outside of HDFS Write

[Ilya Ganelin (JIRA)]

Title: Message Title

Ilya Ganelin created an issue   Apache NiFi / NIFI-3420 NIFI Should support generating Hadoop-readable Lz4 outside of HDFS Write Issue Type: New Feature Assignee: Unassigned Created: 30/Jan/17 19:16 Priority: Major Reporter: Ilya Ganelin Per https://issues.apache.org/jira/browse/HADOOP-12990 data stored in Lz4 format on Hadoop is in a different format from the data generated by the Lz4 CLI. The Lz4 CLI can also not be used to generate the Hadoop-compatible format. At the moment, NiFi does not support compression to Lz4 for streaming data. Although PutHdfs in the Hadoop processors supports writing out Lz4 to HDFS (assuming the appropriate codec exists), if data is instead being saved to something like S3 or simply streamed, there's no way to generate Lz4 compressed data. If the Lz4 command line tool is used within a custom processor to perform Lz4 conversion, this data will then not be readable on Hadoop if it's subsequently loaded to HDFS. A processor can be added that performs the conversion streaming data into the Lz4 format that IS readable on Hadoop by using the Hadoop Lz4 Codec to do the compression. Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)