[
https://issues.apache.org/jira/browse/NIFI-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-16478:
------------------------------
Description:
When a field's schema is a CHOICE between an array of strings and a nested
array of records, nested array values can be written as Java object strings
instead of keeping their structure. For example, this value:
{noformat}
[[[]], [[{"key": "a", "value": "b"}]]]
{noformat}
is written as:
{noformat}
["[[Ljava.lang.Object;@4613311f]", "[[Ljava.lang.Object;@6540cf1d]"]
{noformat}
This happens, for example, when ConsumeKafka uses "Continue with Merged Schema"
and one message has an empty array for a field (inferred as an array of
strings) while another message has nested arrays of records for the same field.
{{DataTypeUtils.chooseDataType}} considers both choices compatible because any
{{Object}} array is accepted as a String. Re-inferring the value's type does
not match either choice, so it falls back to the first compatible choice. When
that choice is the array of strings, each inner array is converted with
{{Arrays.toString}}.
When no choice is a clear match, {{chooseDataType}} should prefer a choice that
does not convert nested arrays, collections, or maps into Strings.
was:
When a field's schema is a CHOICE between an array of strings and a nested
array of records, values such as {{[[[]], [[{"key":"a"}]]]}} can be written as
strings like {{"[[Ljava.lang.Object;@4613311f]"}} instead of keeping their
nested structure. This happens, for example, when ConsumeKafka uses "Continue
with Merged Schema" and one message has an empty array ({{[]}}, inferred as an
array of strings) while another has nested arrays of records for the same field.
{{DataTypeUtils.chooseDataType}} considers both choices compatible because any
{{Object[]}} is accepted as a String. Re-inferring the value's type does not
match either choice, so it falls back to the first compatible choice. When that
choice is the array of strings, each inner array is converted with
{{Arrays.toString}}.
When no choice is a clear match, {{chooseDataType}} should prefer a choice that
does not convert nested arrays, collections, or maps into Strings.
> Nested arrays are written as Java object strings when a CHOICE includes an
> array of strings
> -------------------------------------------------------------------------------------------
>
> Key: NIFI-16478
> URL: https://issues.apache.org/jira/browse/NIFI-16478
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When a field's schema is a CHOICE between an array of strings and a nested
> array of records, nested array values can be written as Java object strings
> instead of keeping their structure. For example, this value:
> {noformat}
> [[[]], [[{"key": "a", "value": "b"}]]]
> {noformat}
> is written as:
> {noformat}
> ["[[Ljava.lang.Object;@4613311f]", "[[Ljava.lang.Object;@6540cf1d]"]
> {noformat}
> This happens, for example, when ConsumeKafka uses "Continue with Merged
> Schema" and one message has an empty array for a field (inferred as an array
> of strings) while another message has nested arrays of records for the same
> field.
> {{DataTypeUtils.chooseDataType}} considers both choices compatible because
> any {{Object}} array is accepted as a String. Re-inferring the value's type
> does not match either choice, so it falls back to the first compatible
> choice. When that choice is the array of strings, each inner array is
> converted with {{Arrays.toString}}.
> When no choice is a clear match, {{chooseDataType}} should prefer a choice
> that does not convert nested arrays, collections, or maps into Strings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
