[jira] [Updated] (NIFI-5591) Enable compression of Avro in ExecuteSQL
[
https://issues.apache.org/jira/browse/NIFI-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Thomsen updated NIFI-5591:
---
Resolution: Fixed
Fix Version/s: 1.8.0
Status: Resolved (was: Patch Available)
> Enable compression of Avro in ExecuteSQL
>
>
> Key: NIFI-5591
> URL: https://issues.apache.org/jira/browse/NIFI-5591
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: macOS, Java 8
>Reporter: Colin Dean
>Assignee: Pierre Villard
>Priority: Major
> Labels: ExecuteSQL, avro, compression
> Fix For: 1.8.0
>
>
> The Avro stream that comes out of the ExecuteSQL processor is uncompressed.
> It's possible to rewrite it compressed using a combination of ConvertRecord
> processor with AvroReader and AvroRecordSetWriter, but that's a lot of extra
> I/O that could be handled transparently at the moment that the Avro data is
> created.
> For implementation, it looks like ExecuteSQL builds a set of
> {{JdbcCommon.AvroConvertionOptions}}[here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java#L246].
> That options object would need to gain a compression flag. Then, within
> {{JdbcCommon#convertToAvroStream}}
> [here|https://github.com/apache/nifi/blob/0dd4a91a6741eec04965a260c8aff38b72b3828d/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java#L281],
> the {{dataFileWriter}} would get a codec set by {{setCodec}}, with the codec
> having been created shortly before.
> For example of creating the codec, I looked at how the AvroRecordSetWriter
> does it. The {{setCodec()}} is performed
> [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/WriteAvroResultWithSchema.java#L44]
> after the codec is created by factory option
> [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L104]
> using a factory method
> [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L137].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5591) Enable compression of Avro in ExecuteSQL
[
https://issues.apache.org/jira/browse/NIFI-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5591:
-
Component/s: (was: Core Framework)
Extensions
> Enable compression of Avro in ExecuteSQL
>
>
> Key: NIFI-5591
> URL: https://issues.apache.org/jira/browse/NIFI-5591
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: macOS, Java 8
>Reporter: Colin Dean
>Assignee: Pierre Villard
>Priority: Major
> Labels: ExecuteSQL, avro, compression
>
> The Avro stream that comes out of the ExecuteSQL processor is uncompressed.
> It's possible to rewrite it compressed using a combination of ConvertRecord
> processor with AvroReader and AvroRecordSetWriter, but that's a lot of extra
> I/O that could be handled transparently at the moment that the Avro data is
> created.
> For implementation, it looks like ExecuteSQL builds a set of
> {{JdbcCommon.AvroConvertionOptions}}[here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java#L246].
> That options object would need to gain a compression flag. Then, within
> {{JdbcCommon#convertToAvroStream}}
> [here|https://github.com/apache/nifi/blob/0dd4a91a6741eec04965a260c8aff38b72b3828d/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java#L281],
> the {{dataFileWriter}} would get a codec set by {{setCodec}}, with the codec
> having been created shortly before.
> For example of creating the codec, I looked at how the AvroRecordSetWriter
> does it. The {{setCodec()}} is performed
> [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/WriteAvroResultWithSchema.java#L44]
> after the codec is created by factory option
> [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L104]
> using a factory method
> [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L137].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5591) Enable compression of Avro in ExecuteSQL
[
https://issues.apache.org/jira/browse/NIFI-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5591:
-
Status: Patch Available (was: Open)
> Enable compression of Avro in ExecuteSQL
>
>
> Key: NIFI-5591
> URL: https://issues.apache.org/jira/browse/NIFI-5591
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: macOS, Java 8
>Reporter: Colin Dean
>Assignee: Pierre Villard
>Priority: Major
> Labels: ExecuteSQL, avro, compression
>
> The Avro stream that comes out of the ExecuteSQL processor is uncompressed.
> It's possible to rewrite it compressed using a combination of ConvertRecord
> processor with AvroReader and AvroRecordSetWriter, but that's a lot of extra
> I/O that could be handled transparently at the moment that the Avro data is
> created.
> For implementation, it looks like ExecuteSQL builds a set of
> {{JdbcCommon.AvroConvertionOptions}}[here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java#L246].
> That options object would need to gain a compression flag. Then, within
> {{JdbcCommon#convertToAvroStream}}
> [here|https://github.com/apache/nifi/blob/0dd4a91a6741eec04965a260c8aff38b72b3828d/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java#L281],
> the {{dataFileWriter}} would get a codec set by {{setCodec}}, with the codec
> having been created shortly before.
> For example of creating the codec, I looked at how the AvroRecordSetWriter
> does it. The {{setCodec()}} is performed
> [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/WriteAvroResultWithSchema.java#L44]
> after the codec is created by factory option
> [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L104]
> using a factory method
> [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L137].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
