[jira] [Updated] (NIFI-5755) Allow PutParquet prosessor to dynamically set config properties for parquet lib
[
https://issues.apache.org/jira/browse/NIFI-5755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Bende updated NIFI-5755:
--
Status: Patch Available (was: Open)
I made this improvement as part of implementing Parquet record reader and
writer, since the record writer needed the same properties being suggested here.
It is included in this PR:
https://github.com/apache/nifi/pull/3679
> Allow PutParquet prosessor to dynamically set config properties for parquet
> lib
> ---
>
> Key: NIFI-5755
> URL: https://issues.apache.org/jira/browse/NIFI-5755
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Ken Tore Tallakstad
>Assignee: Bryan Bende
>Priority: Major
> Attachments: parq2.png
>
>
> PutParquet (NiFi 1.7.1), will fail to write parquet if incoming records
> contain arrays with null elements:
> Schema snippet example:
> { "name": "myfield", "type" : ["null", \\{ "type" :
> "array", "items" : ["null","string"], "default": null }
> ], "default": null
> },
>
> And a corresponding data example:
> "myfield" : [ null, "value1" ],
> "myfield" : [ null, "value2" ],
> "myfield" : [ "value3", null, "value4" ],
>
> Avro does not seem to have a problem with this, but Put parquet fails with
> the following error: "Array contains a null element at X".
>
> There is a parquet config to allow this:
> parquet.avro.write-old-list-structure=false
>
> If the processor would have a box for toggling this value, or a text box to
> pass configs in general would be great!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (NIFI-5755) Allow PutParquet prosessor to dynamically set config properties for parquet lib
[
https://issues.apache.org/jira/browse/NIFI-5755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ken Tore Tallakstad updated NIFI-5755:
--
Description:
PutParquet (NiFi 1.7.1), will fail to write parquet if incoming records contain
arrays with null elements:
Schema snippet example:
{ "name": "myfield", "type" : ["null", \\{ "type" :
"array", "items" : ["null","string"], "default": null }
], "default": null
},
And a corresponding data example:
"myfield" : [ null, "value1" ],
"myfield" : [ null, "value2" ],
"myfield" : [ "value3", null, "value4" ],
Avro does not seem to have a problem with this, but Put parquet fails with the
following error: "Array contains a null element at X".
There is a parquet config to allow this:
parquet.avro.write-old-list-structure=false
If the processor would have a box for toggling this value, or a text box to
pass configs in general would be great!
was:
PutParquet (NiFi 1.7.1), will fail to write parquet if incoming records contain
arrays with null elements:
Schema snippet example:
{
"name": "myfield",
"type" : ["null", \{ "type" : "array", "items" :
["null","string"], "default": null } ], "default": null
},
And a corresponding data example:
"myfield" : [ null, "value1" ],
"myfield" : [ null, "value2" ],
"myfield" : [ "value3", null, "value4" ],
Avro does not seem to have a problem with this, but Put parquet fails with the
following error: "Array contains a null element at X".
There is a parquet config to allow this:
parquet.avro.write-old-list-structure=false
If the processor would have a box for toggling this value, or a text box to
pass configs in general would be great! !parq2.png!
> Allow PutParquet prosessor to dynamically set config properties for parquet
> lib
> ---
>
> Key: NIFI-5755
> URL: https://issues.apache.org/jira/browse/NIFI-5755
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Ken Tore Tallakstad
>Priority: Major
> Attachments: parq2.png
>
>
> PutParquet (NiFi 1.7.1), will fail to write parquet if incoming records
> contain arrays with null elements:
> Schema snippet example:
> { "name": "myfield", "type" : ["null", \\{ "type" :
> "array", "items" : ["null","string"], "default": null }
> ], "default": null
> },
>
> And a corresponding data example:
> "myfield" : [ null, "value1" ],
> "myfield" : [ null, "value2" ],
> "myfield" : [ "value3", null, "value4" ],
>
> Avro does not seem to have a problem with this, but Put parquet fails with
> the following error: "Array contains a null element at X".
>
> There is a parquet config to allow this:
> parquet.avro.write-old-list-structure=false
>
> If the processor would have a box for toggling this value, or a text box to
> pass configs in general would be great!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5755) Allow PutParquet prosessor to dynamically set config properties for parquet lib
[
https://issues.apache.org/jira/browse/NIFI-5755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ken Tore Tallakstad updated NIFI-5755:
--
Attachment: parq2.png
> Allow PutParquet prosessor to dynamically set config properties for parquet
> lib
> ---
>
> Key: NIFI-5755
> URL: https://issues.apache.org/jira/browse/NIFI-5755
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Ken Tore Tallakstad
>Priority: Major
> Attachments: parq2.png
>
>
> PutParquet (NiFi 1.7.1), will fail to write parquet if incoming records
> contain arrays with null elements:
> Schema snippet example:
> { "name": "myfield", "type" : ["null", \\{ "type" :
> "array", "items" : ["null","string"], "default": null }
> ], "default": null
> },
>
> And a corresponding data example:
> "myfield" : [ null, "value1" ],
> "myfield" : [ null, "value2" ],
> "myfield" : [ "value3", null, "value4" ],
>
> Avro does not seem to have a problem with this, but Put parquet fails with
> the following error: "Array contains a null element at X".
>
> There is a parquet config to allow this:
> parquet.avro.write-old-list-structure=false
>
> If the processor would have a box for toggling this value, or a text box to
> pass configs in general would be great!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
