[ https://issues.apache.org/jira/browse/NIFI-7601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dirk Arends updated NIFI-7601: ------------------------------ Description: I am running into an issue QueryRecord processors on Nifi 1.11.4 which was not an issue on 1.11.0. If a record contains arrays of “complex” data I am not able to perform any queries on the record, even if the queries don’t require data from within the array. {code:java} 2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.QueryRecord QueryRecord[id=01731003-15bb-12a6-7e03-8c418ab7fb97] Unable to query StandardFlowFileRecord[uuid=04ed077b-0556-4dda-836e-62c5f9a5c772,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1594014641476-1, container=default, section=1], offset=51521, length=334],offset=0,name=04ed077b-0556-4dda-836e-62c5f9a5c772,size=334] due to java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id: 1111: java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id: 1111 java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id: 1111 at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValue(AbstractCursor.java:1346) at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(AbstractCursor.java:1299) at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(AbstractCursor.java:1352) at org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.java:729) at org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(ResultSetRecordSet.java:186) at org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(ResultSetRecordSet.java:156) at org.apache.nifi.serialization.record.ResultSetRecordSet.<init>(ResultSetRecordSet.java:60) at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:332) at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2746) at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:325) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) {code} [https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/org/apache/calcite/avatica/util/AbstractCursor.java#L1346] I believe this to be a regression, possibly introduced by this commit (though I haven't been able to confirm): [https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077d297] https://issues.apache.org/jira/browse/NIFI-7095 Example 1: Succeeds when performing a query in a QueryRecord Data: {code:java} {"total":5,"payload":["Test"]}{code} Schema: {code:java} {"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[ {"name":"total","type":"int"}, {"name":"payload","type":[ {"type":"array","items":"string"} ]} ]} {code} Example 2: Fails when performing a query in a QueryRecord Data: {code:java} {"total":5,"payload":[{"type":"Test"}]}{code} Schema: {code:java} {"type":"record","name":"ApiResponse","namespace":"com.example.api","fields":[ {"name":"total","type":"int"}, {"name":"payload","type":[ "null", {"type":"array","items":{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[ {"name":"type","type":"string"} ]}} ]} ]} {code} I have set up a simple flow.xml which demonstrates the issue. It is reproducible with a fresh copy of the convenience binaries, with no special steps. was: I am running into an issue QueryRecord processors on Nifi 1.11.4 which was not an issue on 1.11.0. If a record contains arrays of “complex” data I am not able to perform any queries on the record, even if the queries don’t require data from within the array. {code:java} 2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.QueryRecord QueryRecord[id=01731003-15bb-12a6-7e03-8c418ab7fb97] Unable to query StandardFlowFileRecord[uuid=04ed077b-0556-4dda-836e-62c5f9a5c772,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1594014641476-1, container=default, section=1], offset=51521, length=334],offset=0,name=04ed077b-0556-4dda-836e-62c5f9a5c772,size=334] due to java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id: 1111: java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id: 1111 java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id: 1111 at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValue(AbstractCursor.java:1346) at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(AbstractCursor.java:1299) at org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(AbstractCursor.java:1352) at org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.java:729) at org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(ResultSetRecordSet.java:186) at org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(ResultSetRecordSet.java:156) at org.apache.nifi.serialization.record.ResultSetRecordSet.<init>(ResultSetRecordSet.java:60) at org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:332) at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2746) at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:325) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) {code} [https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/org/apache/calcite/avatica/util/AbstractCursor.java#L1346] I believe this to be a regression, possibly introduced by this commit (though I haven't been able to confirm): [https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077d297] https://issues.apache.org/jira/browse/NIFI-7095 Example 1: Succeeds when performing a query in a QueryRecord Data: {code:java} {"total":5,"payload":["Test"]}{code} Schema: {code:java} {"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[ {"name":"total","type":"int"}, {"name":"payload","type":[ {"type":"array","items":"string"} ]} ]} {code} Example 2: Fails when performing a query in a QueryRecord Data: {code:java} {"total":5,"payload":[{"type":"Test"}]}{code} Schema: {code:java} {"type":"record","name":"ApiResponse","namespace":"com.example.api","fields":[ {"name":"total","type":"int"}, {"name":"payload","type":[ "null", {"type":"array","items":{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[ {"name":"type","type":"string"} ]}} ]} ]} {code} I have set up a simple flow.xml which demonstrates the issue. It is reproducible with a fresh copy of the convenience binaries, with no special steps. > QueryRecord failing to handle arrays in 1.11.4 > ---------------------------------------------- > > Key: NIFI-7601 > URL: https://issues.apache.org/jira/browse/NIFI-7601 > Project: Apache NiFi > Issue Type: Bug > Affects Versions: 1.11.4 > Reporter: Dirk Arends > Priority: Major > Attachments: flow.xml > > > I am running into an issue QueryRecord processors on Nifi 1.11.4 which was > not an issue on 1.11.0. If a record contains arrays of “complex” data I am > not able to perform any queries on the record, even if the queries don’t > require data from within the array. > > {code:java} > 2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8] > o.a.nifi.processors.standard.QueryRecord > QueryRecord[id=01731003-15bb-12a6-7e03-8c418ab7fb97] Unable to query > StandardFlowFileRecord[uuid=04ed077b-0556-4dda-836e-62c5f9a5c772,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1594014641476-1, container=default, > section=1], offset=51521, > length=334],offset=0,name=04ed077b-0556-4dda-836e-62c5f9a5c772,size=334] due > to java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, > id: 1111: java.lang.IllegalStateException: Unhandled ARRAY component type: > OBJECT, id: 1111 > java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id: > 1111 > at > org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValue(AbstractCursor.java:1346) > at > org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(AbstractCursor.java:1299) > at > org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(AbstractCursor.java:1352) > at > org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.java:729) > at > org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(ResultSetRecordSet.java:186) > at > org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(ResultSetRecordSet.java:156) > at > org.apache.nifi.serialization.record.ResultSetRecordSet.<init>(ResultSetRecordSet.java:60) > at > org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:332) > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2746) > at > org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:325) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > {code} > [https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/org/apache/calcite/avatica/util/AbstractCursor.java#L1346] > > I believe this to be a regression, possibly introduced by this commit (though > I haven't been able to confirm): > > [https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077d297] > https://issues.apache.org/jira/browse/NIFI-7095 > > Example 1: Succeeds when performing a query in a QueryRecord > > Data: > {code:java} > {"total":5,"payload":["Test"]}{code} > > Schema: > {code:java} > {"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[ > {"name":"total","type":"int"}, > {"name":"payload","type":[ > {"type":"array","items":"string"} > ]} > ]} > {code} > > Example 2: Fails when performing a query in a QueryRecord > > Data: > {code:java} > {"total":5,"payload":[{"type":"Test"}]}{code} > > Schema: > {code:java} > {"type":"record","name":"ApiResponse","namespace":"com.example.api","fields":[ > {"name":"total","type":"int"}, > {"name":"payload","type":[ > "null", > > {"type":"array","items":{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[ > {"name":"type","type":"string"} > ]}} > ]} > ]} > {code} > > I have set up a simple flow.xml which demonstrates the issue. It is > reproducible with a fresh copy of the convenience binaries, with no special > steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)