[jira] [Created] (NIFI-12309) InvokeScriptedProcessor sometimes does not reload script
Dirk Arends created NIFI-12309:
--
Summary: InvokeScriptedProcessor sometimes does not reload script
Key: NIFI-12309
URL: https://issues.apache.org/jira/browse/NIFI-12309
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.22.0
Reporter: Dirk Arends
In the file InvokeScriptedProcessor.java, the function reloadScript should
return the opposite boolean value similar to the return value of the
reloadScriptBody and reloadScriptFile functions in the same file:
{code:java}
- // return whether there were any issues loading the configured script
- return !results.isEmpty();
+ // return whether there were no issues loading the configured script
+ return results.isEmpty();{code}
[https://github.com/apache/nifi/blob/rel/nifi-1.23.2/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/InvokeScriptedProcessor.java#L464]
Going by the function's return value description, the function should return
true if validationResults is empty:
"@return Whether the script was successfully reloaded"
We've been observing InvokeScriptedProcessors intermittently becoming invalid
after a nifi restart which started after nifi version 1.18 where a related
change *by @mattyb149* was made for NIFI-10387:
[https://github.com/apache/nifi/pull/6325]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-6229) support GraalVM
[ https://issues.apache.org/jira/browse/NIFI-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17765491#comment-17765491 ] Dirk Arends commented on NIFI-6229: --- An update on where I've gotten up to with integrating Graal into Nifi: 1. I've managed to mostly work around the issues mapping JS types to Java types, using a combination of target type mappings and adjusting my JS code. Some combinations of JS and Java data types still cause issues. 2. I'm still encountering issues with multithreading when using Graal. Unlike Nashorn, Graal contexts can't be shared among multiple threads. I'm not sure how to handle this, especially in the case of InvokeScriptedProcessor (which invokes the script in order to fetch a list of properties, relationships, etc). Potentially some kind of Graal context pool might be the way to go, but that's not something I've got experience with. If anybody has any suggestions on how to approach multithreading support, they'd be very welcome. > support GraalVM > --- > > Key: NIFI-6229 > URL: https://issues.apache.org/jira/browse/NIFI-6229 > Project: Apache NiFi > Issue Type: Wish >Reporter: Behrouz >Priority: Minor > Labels: features > Attachments: Graal_Testing_Examples.json, Standalone_Graal_Test.java > > > supporting GraalVM ([https://www.graalvm.org/]) would be very helpful for > Nodejs, R, C++, ... programmer to work with Nifi. > "GraalVM is a universal virtual machine for running applications written in > JavaScript, Python, Ruby, R, JVM-based languages like Java, Scala, Kotlin, > Clojure, and LLVM-based languages such as C and C++. > GraalVM removes the isolation between programming languages and enables > interoperability in a shared runtime. It can run either standalone or in the > context of OpenJDK, Node.js, Oracle Database, or MySQL" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-6229) support GraalVM
[
https://issues.apache.org/jira/browse/NIFI-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17728556#comment-17728556
]
Dirk Arends commented on NIFI-6229:
---
I’ve been attempting to get GraalVM working with the latest snapshot (2.0.0) of
NiFi (or any version for that matter) with limited success. The following is a
summary of the progress made in the hope anyone else is able to assist.
*Versions:*
- graalvm-ce-java17-22.3.2
- NiFi 2.0.0-SNAPSHOT (fork)
- Maven 3.9.2 (mvn) [1]
h2. Test setup steps:
1. Download GraalVM from their github released page [2]
2. Install js into the GrallVM [3]
{code:java}
///graalvm-ce-java17-22.3.1/bin/gu install js
{code}
3. Clone my fork of NiFi 2.0.0-SNAPSHOT [4]
4. Build NiFi. In the root dir of the source tree:
{code:java}
export JAVA_HOME=///graalvm-ce-java17-22.3.1
mvn -T 2 clean install -Pinclude-grpc -DskipTests
{code}
5. Add {{java.arg.jsengine}} to
{{./nifi-assembly/target/nifi-2.0.0-SNAPSHOT-bin/nifi-2.0.0-SNAPSHOT/conf.bootstrap.conf}}
{code:java}
java.arg.jsengine=-Dpolyglot.js.nashorn-compat=true
{code}
6. Run NiFi
{code:java}
export JAVA_HOME=///graalvm-ce-java17-22.3.1
./nifi-assembly/target/nifi-2.0.0-SNAPSHOT-bin/nifi-2.0.0-SNAPSHOT/bin/nifi.sh
run
{code}
7. Import the attached flow definition [^Graal_Testing_Examples.json]
h2. Troubleshooting:
If ExecuteScript or InvokeScriptedProcessor processors throw errors saying
there is no script engine, this means either:
1. {{JAVA_HOME}} is not set to GraalVM so NiFi is not running on GraalVM (NiFi
lists the JAVA_HOME target in the terminal when the run command is used)
2. JavaScript is not installed on GraalVM. (see Test setup steps #2)
h2. Test issue 1 - Unsupported operation identifier 'iterator' and object
'[object Array]':
# Complete steps from above
# The sample includes two process groups
## NiFi example scripts - scripts originally from NiFis test scripts [5]
--- test_reader.js
--- test_onTrigger.js
--- testScriptROutesToFailure.js
--- testInvokeScriptCausesException.js
## Minimal sample scripts
--- Basic ExecuteScript
--- Basic InvokeScriptedProcessor
--- Basic InvokeScriptedProcessor (Built with Rollup to allow us to use
TypeScript and newer ES features)
# Each process group loosely has
## Imported scripts with with no changes
## Modified scripts to include a workaround for the following error. I had to
change all occurrences of initialising an array with {{[]}} to {{{}new
Java.type('java.util.ArrayList')(){}}}:
{code:java}
16:14:27 AEST ERROR
InvokeScriptedProcessor[id=053180d8-05e2-311d-ebd7-1665877a86e7] Unable to get
relationships from scripted Processor: org.graalvm.polyglot.PolyglotException:
Unsupported operation identifier 'iterator' and object '[object
Array]'(language: JavaScript, type: Array). Identifier is not executable or
instantiable.
{code}
{code:java}
16:14:26 AEST ERROR
InvokeScriptedProcessor[id=053180d8-05e2-311d-ebd7-1665877a86e7] Unable to
validate the script Processor: java.lang.UnsupportedOperationException:
Unsupported operation identifier 'size' and object '[object Array]'(language:
JavaScript, type: Array). Identifier is not executable or instantiable.:
java.lang.UnsupportedOperationException: Unsupported operation identifier
'size' and object '[object Array]'(language: JavaScript, type: Array).
Identifier is not executable or instantiable.
- Caused by: Attached Guest Language Frames (1)
{code}
I’ve found references to issues with casting GraalVM variables incorrectly
between JavaScript and Java [6]. I was not able to replicate this issue using a
standalone GraalVM script
{code:java}
./graalvm-ce-17/bin/javac Standalone_Graal_Test.java -truffle
./graalvm-ce-17/bin/java Standalone_Graal_Test -Dpolyglot.js.nashorn-compat=true
{code}
[^Standalone_Graal_Test.java]
In the long term, having to use this workaround is not ideal because part of
the appeal to using GraalVM is the newer versions of supported Javascript.
h2. Test issue 2 - Multi threaded access requested by thread but is not allowed
for language(s) js:
Following on from Test Issue #1.3. All scripts with the array fix are now able
to run. Unfortunately they will sometimes (fairly regularly) complain about
multithreading not being allowed.
{code:java}
16:16:14 AEST ERROR
InvokeScriptedProcessor[id=cd0d8770-801f-3790-ac2b-33f47dd0059c] Unable to get
relationships from scripted Processor: java.lang.IllegalStateException: Multi
threaded access requested by thread Thread[NiFi Web Server-152,5,main] but is
not allowed for language(s) js.
{code}
It seems that different NiFi threads are trying to access the script object
simultaneously which seems to indicate a larger underlying issue with the
differences between Nashorn and GraalVM. While some solutions are documented
I'm not sure how they can apply to NiFi.
{quote}the main reason why we do not allow two concurrent threads to acce
[jira] [Updated] (NIFI-6229) support GraalVM
[ https://issues.apache.org/jira/browse/NIFI-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dirk Arends updated NIFI-6229: -- Attachment: Standalone_Graal_Test.java > support GraalVM > --- > > Key: NIFI-6229 > URL: https://issues.apache.org/jira/browse/NIFI-6229 > Project: Apache NiFi > Issue Type: Wish >Reporter: Behrouz >Priority: Minor > Labels: features > Attachments: Graal_Testing_Examples.json, Standalone_Graal_Test.java > > > supporting GraalVM ([https://www.graalvm.org/]) would be very helpful for > Nodejs, R, C++, ... programmer to work with Nifi. > "GraalVM is a universal virtual machine for running applications written in > JavaScript, Python, Ruby, R, JVM-based languages like Java, Scala, Kotlin, > Clojure, and LLVM-based languages such as C and C++. > GraalVM removes the isolation between programming languages and enables > interoperability in a shared runtime. It can run either standalone or in the > context of OpenJDK, Node.js, Oracle Database, or MySQL" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-6229) support GraalVM
[ https://issues.apache.org/jira/browse/NIFI-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dirk Arends updated NIFI-6229: -- Attachment: Graal_Testing_Examples.json > support GraalVM > --- > > Key: NIFI-6229 > URL: https://issues.apache.org/jira/browse/NIFI-6229 > Project: Apache NiFi > Issue Type: Wish >Reporter: Behrouz >Priority: Minor > Labels: features > Attachments: Graal_Testing_Examples.json > > > supporting GraalVM ([https://www.graalvm.org/]) would be very helpful for > Nodejs, R, C++, ... programmer to work with Nifi. > "GraalVM is a universal virtual machine for running applications written in > JavaScript, Python, Ruby, R, JVM-based languages like Java, Scala, Kotlin, > Clojure, and LLVM-based languages such as C and C++. > GraalVM removes the isolation between programming languages and enables > interoperability in a shared runtime. It can run either standalone or in the > context of OpenJDK, Node.js, Oracle Database, or MySQL" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-7601) QueryRecord failing to handle arrays
[
https://issues.apache.org/jira/browse/NIFI-7601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17170455#comment-17170455
]
Dirk Arends commented on NIFI-7601:
---
[~mattyb149] Can I provide any further assistance with tests or other details
for this issue? I'm not sure where to start for a PR unfortunately, but it's
impacting my usage pretty severely. I've had to downgrade to 1.11.1 and not
being able to upgrade to 1.12 will mean missing out on a couple of other PRs I
helped with that have been merged. If there's any other support I can provide
I'd be happy to do whatever I can!
> QueryRecord failing to handle arrays
>
>
> Key: NIFI-7601
> URL: https://issues.apache.org/jira/browse/NIFI-7601
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.11.4
>Reporter: Dirk Arends
>Assignee: Matt Burgess
>Priority: Major
> Attachments: flow.xml
>
>
> I am running into an issue QueryRecord processors on Nifi 1.11.4 which was
> not an issue on 1.11.0. If a record contains arrays of “complex” data I am
> not able to perform any queries on the record, even if the queries don’t
> require data from within the array.
>
> {code:java}
> 2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8]
> o.a.nifi.processors.standard.QueryRecord
> QueryRecord[id=01731003-15bb-12a6-7e03-8c418ab7fb97] Unable to query
> StandardFlowFileRecord[uuid=04ed077b-0556-4dda-836e-62c5f9a5c772,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1594014641476-1, container=default,
> section=1], offset=51521,
> length=334],offset=0,name=04ed077b-0556-4dda-836e-62c5f9a5c772,size=334] due
> to java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT,
> id: : java.lang.IllegalStateException: Unhandled ARRAY component type:
> OBJECT, id:
> java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id:
>
> at
> org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValue(AbstractCursor.java:1346)
> at
> org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(AbstractCursor.java:1299)
> at
> org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(AbstractCursor.java:1352)
> at
> org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.java:729)
> at
> org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(ResultSetRecordSet.java:186)
> at
> org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(ResultSetRecordSet.java:156)
> at
> org.apache.nifi.serialization.record.ResultSetRecordSet.(ResultSetRecordSet.java:60)
> at
> org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:332)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2746)
> at
> org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:325)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> [https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/org/apache/calcite/avatica/util/AbstractCursor.java#L1346]
>
> I believe this to be a regression, possibly introduced by this commit (though
> I haven't been able to confirm):
>
> [https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077d297]
> https://issues.apache.org/jira/browse/NIFI-7095
>
> Example 1: Succeeds when performing a query in a QueryRecord
>
> Data:
> {code:java}
> {"total":5,"payload":["Test"]}{code}
>
> Schema:
> {code:java}
> {"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
> {"name":"total","type":"int"},
> {"name":"payload","type":[
> {"type":"array","items":"string"}
> ]}
> ]}
> {code}
>
> Example 2: Fails when performing a query in a QueryRecord
>
> Data:
> {code:java}
> {"total":5,"payload":[{"ty
[jira] [Updated] (NIFI-6332) PutS3 Processor adding Cache-Control property
[ https://issues.apache.org/jira/browse/NIFI-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dirk Arends updated NIFI-6332: -- Description: The PutS3 Processor does not allow you to set the Cache-Control header. My use case is to be able to control the Cache-Control header on objects intended to be served via S3 website hosting. After looking at other issues I found NIFI-2810 which references [Github PR 1034|https://github.com/apache/nifi/pull/1034] which added the property Content-Type to the PutS3Object. was: The PutS3 Processor does not allow you to set the Cache-Control header. My use case is to be able to control the Cache-Control header objects intended to be served via S3 website hosting. After looking at other issues I found NIFI-2810 which references [Github PR 1034|https://github.com/apache/nifi/pull/1034] which added the property Content-Type to the PutS3Object. > PutS3 Processor adding Cache-Control property > -- > > Key: NIFI-6332 > URL: https://issues.apache.org/jira/browse/NIFI-6332 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Dirk Arends >Priority: Major > > The PutS3 Processor does not allow you to set the Cache-Control header. > My use case is to be able to control the Cache-Control header on objects > intended to be served via S3 website hosting. After looking at other issues I > found NIFI-2810 which references [Github PR > 1034|https://github.com/apache/nifi/pull/1034] which added the property > Content-Type to the PutS3Object. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7664) PutS3 Processor adding Content-Disposition property
[ https://issues.apache.org/jira/browse/NIFI-7664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dirk Arends updated NIFI-7664: -- Description: The PutS3 Processor does not allow you to set the Content-Disposition header. My use case is to be able to control the Content-Disposition header on objects intended to be served via S3 website hosting. This is related to NIFI-6332 and both changes can be use NIFI-2810 as an example of how to implement the changes as NIFI-2810 adds the property Content-Type to the PutS3Object. was: The PutS3 Processor does not allow you to set the Content-Disposition header. My use case is to be able to control the Content-Disposition header objects intended to be served via S3 website hosting. This is related to NIFI-6332 and both changes can be use NIFI-2810 as an example of how to implement the changes as NIFI-2810 adds the property Content-Type to the PutS3Object. > PutS3 Processor adding Content-Disposition property > --- > > Key: NIFI-7664 > URL: https://issues.apache.org/jira/browse/NIFI-7664 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Dirk Arends >Priority: Major > > The PutS3 Processor does not allow you to set the Content-Disposition header. > My use case is to be able to control the Content-Disposition header on > objects intended to be served via S3 website hosting. > This is related to NIFI-6332 and both changes can be use NIFI-2810 as an > example of how to implement the changes as NIFI-2810 adds the property > Content-Type to the PutS3Object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7664) PutS3 Processor adding Content-Disposition property
[ https://issues.apache.org/jira/browse/NIFI-7664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dirk Arends updated NIFI-7664: -- Description: The PutS3 Processor does not allow you to set the Content-Disposition header. My use case is to be able to control the Content-Disposition header objects intended to be served via S3 website hosting. This is related to NIFI-6332 and both changes can be use NIFI-2810 as an example of how to implement the changes as NIFI-2810 adds the property Content-Type to the PutS3Object. was: The PutS3 Processor does not allow you to set the Content-Disposition header. My use case is to be able to control the Cache-Control header objects intended to be served via S3 website hosting. This is related to NIFI-6332 and both changes can be use NIFI-2810 as an example of how to implement the changes as NIFI-2810 adds the property Content-Type to the PutS3Object. > PutS3 Processor adding Content-Disposition property > --- > > Key: NIFI-7664 > URL: https://issues.apache.org/jira/browse/NIFI-7664 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Dirk Arends >Priority: Major > > The PutS3 Processor does not allow you to set the Content-Disposition header. > My use case is to be able to control the Content-Disposition header objects > intended to be served via S3 website hosting. > This is related to NIFI-6332 and both changes can be use NIFI-2810 as an > example of how to implement the changes as NIFI-2810 adds the property > Content-Type to the PutS3Object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7664) PutS3 Processor adding Content-Disposition property
Dirk Arends created NIFI-7664: - Summary: PutS3 Processor adding Content-Disposition property Key: NIFI-7664 URL: https://issues.apache.org/jira/browse/NIFI-7664 Project: Apache NiFi Issue Type: Improvement Reporter: Dirk Arends The PutS3 Processor does not allow you to set the Content-Disposition header. My use case is to be able to control the Cache-Control header objects intended to be served via S3 website hosting. This is related to NIFI-6332 and both changes can be use NIFI-2810 as an example of how to implement the changes as NIFI-2810 adds the property Content-Type to the PutS3Object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7601) QueryRecord failing to handle arrays in 1.11.4
[
https://issues.apache.org/jira/browse/NIFI-7601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dirk Arends updated NIFI-7601:
--
Description:
I am running into an issue QueryRecord processors on Nifi 1.11.4 which was not
an issue on 1.11.0. If a record contains arrays of “complex” data I am not able
to perform any queries on the record, even if the queries don’t require data
from within the array.
{code:java}
2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8]
o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=01731003-15bb-12a6-7e03-8c418ab7fb97] Unable to query
StandardFlowFileRecord[uuid=04ed077b-0556-4dda-836e-62c5f9a5c772,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1594014641476-1, container=default,
section=1], offset=51521,
length=334],offset=0,name=04ed077b-0556-4dda-836e-62c5f9a5c772,size=334] due to
java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id:
: java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT,
id:
java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id:
at
org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValue(AbstractCursor.java:1346)
at
org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(AbstractCursor.java:1299)
at
org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(AbstractCursor.java:1352)
at
org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.java:729)
at
org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(ResultSetRecordSet.java:186)
at
org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(ResultSetRecordSet.java:156)
at
org.apache.nifi.serialization.record.ResultSetRecordSet.(ResultSetRecordSet.java:60)
at
org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:332)
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2746)
at
org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:325)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
[https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/org/apache/calcite/avatica/util/AbstractCursor.java#L1346]
I believe this to be a regression, possibly introduced by this commit (though I
haven't been able to confirm):
[https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077d297]
https://issues.apache.org/jira/browse/NIFI-7095
Example 1: Succeeds when performing a query in a QueryRecord
Data:
{code:java}
{"total":5,"payload":["Test"]}{code}
Schema:
{code:java}
{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
{"name":"total","type":"int"},
{"name":"payload","type":[
{"type":"array","items":"string"}
]}
]}
{code}
Example 2: Fails when performing a query in a QueryRecord
Data:
{code:java}
{"total":5,"payload":[{"type":"Test"}]}{code}
Schema:
{code:java}
{"type":"record","name":"ApiResponse","namespace":"com.example.api","fields":[
{"name":"total","type":"int"},
{"name":"payload","type":[
"null",
{"type":"array","items":{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
{"name":"type","type":"string"}
]}}
]}
]}
{code}
I have set up a simple flow.xml which demonstrates the issue. It is
reproducible with a fresh copy of the convenience binaries, with no special
steps.
was:
I am running into an issue QueryRecord processors on Nifi 1.11.4 which was not
an issue on 1.11.0. If a record contains arrays of “complex” data I am not able
to perform any queries on the record, even if the queries don’t require data
from within the array.
{code:java}
2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8]
o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=01731003-15bb-12a6-7e03-8c418ab7fb97] Unable to query
StandardFlowFileRecord[uuid=04ed077b-0556-4dda-836e-62c5f9
[jira] [Created] (NIFI-7601) QueryRecord failing to handle arrays in 1.11.4
Dirk Arends created NIFI-7601:
-
Summary: QueryRecord failing to handle arrays in 1.11.4
Key: NIFI-7601
URL: https://issues.apache.org/jira/browse/NIFI-7601
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.11.4
Reporter: Dirk Arends
Attachments: flow.xml
I am running into an issue QueryRecord processors on Nifi 1.11.4 which was not
an issue on 1.11.0. If a record contains arrays of “complex” data I am not able
to perform any queries on the record, even if the queries don’t require data
from within the array.
{code:java}
2020-07-06 16:43:53,902 ERROR [Timer-Driven Process Thread-8]
o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=01731003-15bb-12a6-7e03-8c418ab7fb97] Unable to query
StandardFlowFileRecord[uuid=04ed077b-0556-4dda-836e-62c5f9a5c772,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1594014641476-1, container=default,
section=1], offset=51521,
length=334],offset=0,name=04ed077b-0556-4dda-836e-62c5f9a5c772,size=334] due to
java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id:
: java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT,
id:
java.lang.IllegalStateException: Unhandled ARRAY component type: OBJECT, id:
at
org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.convertValue(AbstractCursor.java:1346)
at
org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getObject(AbstractCursor.java:1299)
at
org.apache.calcite.avatica.util.AbstractCursor$ArrayAccessor.getArray(AbstractCursor.java:1352)
at
org.apache.calcite.avatica.AvaticaResultSet.getArray(AvaticaResultSet.java:729)
at
org.apache.nifi.serialization.record.ResultSetRecordSet.getDataType(ResultSetRecordSet.java:186)
at
org.apache.nifi.serialization.record.ResultSetRecordSet.createSchema(ResultSetRecordSet.java:156)
at
org.apache.nifi.serialization.record.ResultSetRecordSet.(ResultSetRecordSet.java:60)
at
org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:332)
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2746)
at
org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:325)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
[https://github.com/apache/calcite-avatica/blob/master/core/src/main/java/org/apache/calcite/avatica/util/AbstractCursor.java#L1346]
I believe this to be a regression, possibly introduced by this commit (though I
haven't been able to confirm):
[https://github.com/apache/nifi/commit/98f9b7c033a8d80ddf43aa77f63107703077d297]
https://issues.apache.org/jira/browse/NIFI-7095
Example 1: Succeeds when performing a query in a QueryRecord
Data:
{code:java}
{"total":5,"payload":["Test"]}{code}
Schema:
{code:java}
{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
{"name":"total","type":"int"},
{"name":"payload","type":[
{"type":"array","items":"string"}
]}
]}
{code}
Example 2: Fails when performing a query in a QueryRecord
Data:
{code:java}
{"total":5,"payload":[{"type":"Test"}]}{code}
Schema:
{code:java}
{"type":"record","name":"ApiResponse","namespace":"com.example.api","fields":[
{"name":"total","type":"int"},
{"name":"payload","type":[
"null",
{"type":"array","items":{"type":"record","name":"ApiResponseData","namespace":"com.example.api","fields":[
{"name":"type","type":"string"}
]}}
]}
]}
{code}
I have set up a simple flow.xml which demonstrates the issue. It is
reproducible with a fresh copy of the convenience binaries, with no special
steps.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-6332) PutS3 Processor adding Cache-Control property
[ https://issues.apache.org/jira/browse/NIFI-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dirk Arends updated NIFI-6332: -- Description: The PutS3 Processor does not allow you to set the Cache-Control header. My use case is to be able to control the Cache-Control header objects intended to be served via S3 website hosting. After looking at other issues I found NIFI-2810 which references [Github PR 1034|https://github.com/apache/nifi/pull/1034] which added the property Content-Type to the PutS3Object. was: The PutS3 Processor does not allow you to set the Cache-Control header. My use case is to put files into S3 to be served and cached via S3 website hosting. After looking at other issues I found NIFI-2810 which references [Github PR 1034|https://github.com/apache/nifi/pull/1034] which added the property Content-Type to the PutS3Object. Would there be anyone who is able to make these changes. > PutS3 Processor adding Cache-Control property > -- > > Key: NIFI-6332 > URL: https://issues.apache.org/jira/browse/NIFI-6332 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Dirk Arends >Priority: Major > > The PutS3 Processor does not allow you to set the Cache-Control header. > My use case is to be able to control the Cache-Control header objects > intended to be served via S3 website hosting. After looking at other issues I > found NIFI-2810 which references [Github PR > 1034|https://github.com/apache/nifi/pull/1034] which added the property > Content-Type to the PutS3Object. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (NIFI-6332) PutS3 Processor adding Cache-Control property
[ https://issues.apache.org/jira/browse/NIFI-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dirk Arends updated NIFI-6332: -- Description: The PutS3 Processor does not allow you to set the Cache-Control header. My use case is to put files into S3 to be served and cached via S3 website hosting. After looking at other issues I found NIFI-2810 which references [Github PR 1034|https://github.com/apache/nifi/pull/1034] which added the property Content-Type to the PutS3Object. Would there be anyone who is able to make these changes. was: The PutS3 Processor does not allow you to set the Cache-Control header. My use case is to put files into S3 to be served and cached via S3 website hosting. After looking at other issues I found NIFI-2810 which references [Github PR 1034|[https://github.|https://github.com/apache/nifi/pull/1034] [com/apache/nifi/pull/1034|https://github.com/apache/nifi/pull/1034]] which added the property Content-Type to the PutS3Object. Would there be anyone who is able to make these changes. > PutS3 Processor adding Cache-Control property > -- > > Key: NIFI-6332 > URL: https://issues.apache.org/jira/browse/NIFI-6332 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Dirk Arends >Priority: Major > > The PutS3 Processor does not allow you to set the Cache-Control header. My > use case is to put files into S3 to be served and cached via S3 website > hosting. After looking at other issues I found NIFI-2810 which references > [Github PR 1034|https://github.com/apache/nifi/pull/1034] which added the > property Content-Type to the PutS3Object. > > Would there be anyone who is able to make these changes. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (NIFI-6332) PutS3 Processor adding Cache-Control property
[ https://issues.apache.org/jira/browse/NIFI-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dirk Arends updated NIFI-6332: -- Description: The PutS3 Processor does not allow you to set the Cache-Control header. My use case is to put files into S3 to be served and cached via S3 website hosting. After looking at other issues I found NIFI-2810 which references [Github PR 1034|[https://github.|https://github.com/apache/nifi/pull/1034] [com/apache/nifi/pull/1034|https://github.com/apache/nifi/pull/1034]] which added the property Content-Type to the PutS3Object. Would there be anyone who is able to make these changes. was: The PutS3 Processor does not allow you to set the Cache-Control. My use case is to put files into S3 to be served and cached via Cloudflare. After looking at other issues I found NIFI-2810 which references [Github PR 1034|[https://github.|https://github.com/apache/nifi/pull/1034] [com/apache/nifi/pull/1034|https://github.com/apache/nifi/pull/1034]] which added the property Content-Type to PutS3Object. Would there be anyone who and make these changes. > PutS3 Processor adding Cache-Control property > -- > > Key: NIFI-6332 > URL: https://issues.apache.org/jira/browse/NIFI-6332 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Dirk Arends >Priority: Major > > The PutS3 Processor does not allow you to set the Cache-Control header. My > use case is to put files into S3 to be served and cached via S3 website > hosting. After looking at other issues I found NIFI-2810 which references > [Github PR 1034|[https://github.|https://github.com/apache/nifi/pull/1034] > [com/apache/nifi/pull/1034|https://github.com/apache/nifi/pull/1034]] which > added the property Content-Type to the PutS3Object. > > Would there be anyone who is able to make these changes. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (NIFI-6332) PutS3 Processor adding Cache-Control property
[ https://issues.apache.org/jira/browse/NIFI-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dirk Arends updated NIFI-6332: -- Description: The PutS3 Processor does not allow you to set the Cache-Control. My use case is to put files into S3 to be served and cached via Cloudflare. After looking at other issues I found NIFI-2810 which references [Github PR 1034|[https://github.|https://github.com/apache/nifi/pull/1034] [com/apache/nifi/pull/1034|https://github.com/apache/nifi/pull/1034]] which added the property Content-Type to PutS3Object. Would there be anyone who and make these changes. was: The PutS3 Processor does not allow you to set the Cache-Control. My use case is to put files into S3 to be served and cached via Cloudflare. After looking at other issues I found [NIFI-2810|https://issues.apache.org/jira/browse/NIFI-2810] which references [Github PR 1034|[https://github.com/apache/nifi/pull/1034]] which added the property Content-Type to PutS3Object. Would there be anyone who and make these changes. > PutS3 Processor adding Cache-Control property > -- > > Key: NIFI-6332 > URL: https://issues.apache.org/jira/browse/NIFI-6332 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Dirk Arends >Priority: Major > > The PutS3 Processor does not allow you to set the Cache-Control. My use case > is to put files into S3 to be served and cached via Cloudflare. After looking > at other issues I found NIFI-2810 which references [Github PR > 1034|[https://github.|https://github.com/apache/nifi/pull/1034] > [com/apache/nifi/pull/1034|https://github.com/apache/nifi/pull/1034]] which > added the property Content-Type to PutS3Object. > > Would there be anyone who and make these changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-6332) PutS3 Processor adding Cache-Control property
Dirk Arends created NIFI-6332: - Summary: PutS3 Processor adding Cache-Control property Key: NIFI-6332 URL: https://issues.apache.org/jira/browse/NIFI-6332 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.8.0 Reporter: Dirk Arends The PutS3 Processor does not allow you to set the Cache-Control. My use case is to put files into S3 to be served and cached via Cloudflare. After looking at other issues I found [NIFI-2810|https://issues.apache.org/jira/browse/NIFI-2810] which references [Github PR 1034|[https://github.com/apache/nifi/pull/1034]] which added the property Content-Type to PutS3Object. Would there be anyone who and make these changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4957) Enable JoltTransformJSON to pickup a Jolt Spec file from a file location
[ https://issues.apache.org/jira/browse/NIFI-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847246#comment-16847246 ] Dirk Arends commented on NIFI-4957: --- [~rahst12] Hoping this might be close. I have been considering attempting a Nifi processor solution (using FetchFile among others) which wouldn't be anywhere near as clean as a built in Jolt Body attribute. It sounds ideal! > Enable JoltTransformJSON to pickup a Jolt Spec file from a file location > > > Key: NIFI-4957 > URL: https://issues.apache.org/jira/browse/NIFI-4957 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Ryan Hendrickson >Assignee: Ryan Hendrickson >Priority: Minor > Attachments: image-2018-03-09-23-56-43-912.png > > > Add a property to allow the Jolt Spec to be read from a file on disk and/or > the classpath. > !image-2018-03-09-23-56-43-912.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
