[jira] [Assigned] (NIFI-13549) Provenance Lineage's 'Go To' does not always redirect.
[ https://issues.apache.org/jira/browse/NIFI-13549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-13549: Assignee: Nissim Shiman > Provenance Lineage's 'Go To' does not always redirect. > -- > > Key: NIFI-13549 > URL: https://issues.apache.org/jira/browse/NIFI-13549 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.27.0 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > Provenance lineage's option of "Go To" is not redirecting to the processor. > This is only occurring when nifi is part of a cluster. > This has only been observed on 1.x. > To see issue: > Global Menu -> Data Provenance -> Show Lineage icon (of any event) -> right > click on any circle -> choose "Go To" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13549) Provenance Lineage's 'Go To' does not always redirect.
Nissim Shiman created NIFI-13549: Summary: Provenance Lineage's 'Go To' does not always redirect. Key: NIFI-13549 URL: https://issues.apache.org/jira/browse/NIFI-13549 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.27.0 Reporter: Nissim Shiman Provenance lineage's option of "Go To" is not redirecting to the processor. This is only occurring when nifi is part of a cluster. This has only been observed on 1.x. To see issue: Global Menu -> Data Provenance -> Show Lineage icon (of any event) -> right click on any circle -> choose "Go To" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-13486) Outer versioned PG will not indicate modification has occurred when inner versioned PG has been modified
[ https://issues.apache.org/jira/browse/NIFI-13486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17862872#comment-17862872 ] Nissim Shiman commented on NIFI-13486: -- Thank you for commenting. I am seeing now that handling the various permutations of this is heading down a rabbit-hole, so I understand letting this be. > Outer versioned PG will not indicate modification has occurred when inner > versioned PG has been modified > - > > Key: NIFI-13486 > URL: https://issues.apache.org/jira/browse/NIFI-13486 > Project: Apache NiFi > Issue Type: Bug > Components: NiFi Registry >Affects Versions: 1.27.0 >Reporter: Nissim Shiman >Priority: Major > > When a versioned PG is contained in another versioned PG, modifying the inner > one will result in the gui showing the '*' icon for the inner PG. > The outer versioned PG will still have the green checkmark though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-13491) InvokeHTTP new property - Response Header Request Attributes Prefix
[ https://issues.apache.org/jira/browse/NIFI-13491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-13491: Assignee: Nissim Shiman > InvokeHTTP new property - Response Header Request Attributes Prefix > --- > > Key: NIFI-13491 > URL: https://issues.apache.org/jira/browse/NIFI-13491 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.25.0, 2.0.0-M3 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > InvokeHTTP has a property, _Response Header Request Attributes Enabled_ which > is false by default. > When it it set to true, the header request attributes are sent back and > included as flowfile attributes on the _Original_ relationship, but there is > nothing that indicates that the attributes refer to the response (as opposed > to the flowfile contents that were sent). > There will be new attributes of _Content-Type_ and _Content-Length_ with > values that may be confusing to downstream users as they refer to the > response, as opposed to the actual flowfile contents. > This property will allow a prefix to be set, so that flow users will better > understand that these refer to the http response. > This option will only appear if _Response Header Request Attributes Enabled_ > is true. > By default there will be no default value to allow backward compatibility. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13491) InvokeHTTP new property - Response Header Request Attributes Prefix
Nissim Shiman created NIFI-13491: Summary: InvokeHTTP new property - Response Header Request Attributes Prefix Key: NIFI-13491 URL: https://issues.apache.org/jira/browse/NIFI-13491 Project: Apache NiFi Issue Type: Improvement Affects Versions: 2.0.0-M3, 1.25.0 Reporter: Nissim Shiman InvokeHTTP has a property, _Response Header Request Attributes Enabled_ which is false by default. When it it set to true, the header request attributes are sent back and included as flowfile attributes on the _Original_ relationship, but there is nothing that indicates that the attributes refer to the response (as opposed to the flowfile contents that were sent). There will be new attributes of _Content-Type_ and _Content-Length_ with values that may be confusing to downstream users as they refer to the response, as opposed to the actual flowfile contents. This property will allow a prefix to be set, so that flow users will better understand that these refer to the http response. This option will only appear if _Response Header Request Attributes Enabled_ is true. By default there will be no default value to allow backward compatibility. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13486) Outer versioned PG will not indicate modification has occurred when inner versioned PG has been modified
Nissim Shiman created NIFI-13486: Summary: Outer versioned PG will not indicate modification has occurred when inner versioned PG has been modified Key: NIFI-13486 URL: https://issues.apache.org/jira/browse/NIFI-13486 Project: Apache NiFi Issue Type: Bug Components: NiFi Registry Affects Versions: 1.27.0 Reporter: Nissim Shiman When a versioned PG is contained in another versioned PG, modifying the inner one will result in the gui showing the '*' icon for the inner PG. The outer versioned PG will still have the green checkmark though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13420) /system-diagnostics rest call can return a negative number for nonHeapUtilization
Nissim Shiman created NIFI-13420: Summary: /system-diagnostics rest call can return a negative number for nonHeapUtilization Key: NIFI-13420 URL: https://issues.apache.org/jira/browse/NIFI-13420 Project: Apache NiFi Issue Type: Bug Affects Versions: 2.0.0-M2, 1.25.0 Reporter: Nissim Shiman This is seen more clearly in clusters as single node nifis don't always return this in results. Can also be seen in 1.x ui, and old 2.0.0 ui (for clusters) GlobalMenu -> Summary -> system diagnostics -> JVM -> Non Heap (with percentage next to it - this is the number that could be negative) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-13031) /flow/process-groups/{id}/status rest api call will return aggregate metrics for connections for one of the nodes, even when nodewise query parameter is set to true
[
https://issues.apache.org/jira/browse/NIFI-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-13031:
-
Labels: backport-needed (was: )
Status: Patch Available (was: Open)
> /flow/process-groups/{id}/status rest api call will return aggregate metrics
> for connections for one of the nodes, even when nodewise query parameter is
> set to true
>
>
> Key: NIFI-13031
> URL: https://issues.apache.org/jira/browse/NIFI-13031
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 2.0.0-M2, 1.24.0
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Labels: backport-needed
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The rest api call:
> /flow/process-groups/\{id}/status (with nodewise set to true)
> will return aggregate metrics for connections found in the process group as
> well as nodewise metrics for those connections as well.
> One of the nodewise connections will be the same as the aggregate (i.e.
> instead of just reporting the metrics for the connection on that node, it
> will report the metrics across all nodes for that connection). The other
> nodewise connections will report what the connection status is for that node.
> Behavior should be modified so all nodewise connection reporting is just for
> that node.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13031) /flow/process-groups/{id}/status rest api call will return aggregate metrics for connections for one of the nodes, even when nodewise query parameter is set to true
[
https://issues.apache.org/jira/browse/NIFI-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-13031:
-
Description:
The rest api call:
/flow/process-groups/\{id}/status (with nodewise set to true)
will return aggregate metrics for connections found in the process group as
well as nodewise metrics for those connections as well.
One of the nodewise connections will be the same as the aggregate (i.e. instead
of just reporting the metrics for the connection on that node, it will report
the metrics across all nodes for that connection). The other nodewise
connections will report what the connection status is for that node.
Behavior should be modified so all nodewise connection reporting is just for
that node.
was:
The rest api call:
/flow/process-groups/{id}/status (with nodewise set to true) will return
aggregate metrics for connections found in the process group as well as
nodewise metrics for those connections as well.
One of the nodewise connections will be the same as the aggregate (i.e. instead
of just reporting the metrics for the connection on that node, it will report
the metrics across all nodes for that connection). The other nodewise
connections will report what the connection status is for that node.
Behavior should be modified so all nodewise connection reporting is just for
that node.
> /flow/process-groups/{id}/status rest api call will return aggregate metrics
> for connections for one of the nodes, even when nodewise query parameter is
> set to true
>
>
> Key: NIFI-13031
> URL: https://issues.apache.org/jira/browse/NIFI-13031
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
>
> The rest api call:
> /flow/process-groups/\{id}/status (with nodewise set to true)
> will return aggregate metrics for connections found in the process group as
> well as nodewise metrics for those connections as well.
> One of the nodewise connections will be the same as the aggregate (i.e.
> instead of just reporting the metrics for the connection on that node, it
> will report the metrics across all nodes for that connection). The other
> nodewise connections will report what the connection status is for that node.
> Behavior should be modified so all nodewise connection reporting is just for
> that node.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-13031) /flow/process-groups/{id}/status rest api call will return aggregate metrics for connections for one of the nodes, even when nodewise query parameter is set to true
[
https://issues.apache.org/jira/browse/NIFI-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman reassigned NIFI-13031:
Assignee: Nissim Shiman
> /flow/process-groups/{id}/status rest api call will return aggregate metrics
> for connections for one of the nodes, even when nodewise query parameter is
> set to true
>
>
> Key: NIFI-13031
> URL: https://issues.apache.org/jira/browse/NIFI-13031
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
>
> The rest api call:
> /flow/process-groups/{id}/status (with nodewise set to true) will return
> aggregate metrics for connections found in the process group as well as
> nodewise metrics for those connections as well.
> One of the nodewise connections will be the same as the aggregate (i.e.
> instead of just reporting the metrics for the connection on that node, it
> will report the metrics across all nodes for that connection). The other
> nodewise connections will report what the connection status is for that node.
> Behavior should be modified so all nodewise connection reporting is just for
> that node.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-13031) /flow/process-groups/{id}/status rest api call will return aggregate metrics for connections for one of the nodes, even when nodewise query parameter is set to true
Nissim Shiman created NIFI-13031:
Summary: /flow/process-groups/{id}/status rest api call will
return aggregate metrics for connections for one of the nodes, even when
nodewise query parameter is set to true
Key: NIFI-13031
URL: https://issues.apache.org/jira/browse/NIFI-13031
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.0.0-M2, 1.24.0
Reporter: Nissim Shiman
The rest api call:
/flow/process-groups/{id}/status (with nodewise set to true) will return
aggregate metrics for connections found in the process group as well as
nodewise metrics for those connections as well.
One of the nodewise connections will be the same as the aggregate (i.e. instead
of just reporting the metrics for the connection on that node, it will report
the metrics across all nodes for that connection). The other nodewise
connections will report what the connection status is for that node.
Behavior should be modified so all nodewise connection reporting is just for
that node.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-13009) On disconnected node, new flows are removed when reattached to cluster. Add documentation and/or logging for this
[ https://issues.apache.org/jira/browse/NIFI-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-13009: - Affects Version/s: 2.0.0-M2 > On disconnected node, new flows are removed when reattached to cluster. Add > documentation and/or logging for this > -- > > Key: NIFI-13009 > URL: https://issues.apache.org/jira/browse/NIFI-13009 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 2.0.0-M2 >Reporter: Nissim Shiman >Priority: Major > > Disconnected nifi nodes can be modified with new flows even though they are > disconnected. > When the node rejoins the cluster, however, these flows can be quietly > removed. > For example, on a 3 node cluster graph with a blank graph, disconnect one of > the nodes. > On the disconnected node, set up and start: > GenerateFlowFile -> UpdateAttribute (where UpdateAttribute has success > relationship terminated) > On gui of one of the remaining connected nodes, reconnect disconnected node. > It will reconnect successfully and all node graphs will be blank. > Cluster docs and/or logs should capture this situation better. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-13009) On disconnected node, new flows are removed when reattached to cluster. Add documentation and/or logging for this
Nissim Shiman created NIFI-13009: Summary: On disconnected node, new flows are removed when reattached to cluster. Add documentation and/or logging for this Key: NIFI-13009 URL: https://issues.apache.org/jira/browse/NIFI-13009 Project: Apache NiFi Issue Type: Improvement Reporter: Nissim Shiman Disconnected nifi nodes can be modified with new flows even though they are disconnected. When the node rejoins the cluster, however, these flows can be quietly removed. For example, on a 3 node cluster graph with a blank graph, disconnect one of the nodes. On the disconnected node, set up and start: GenerateFlowFile -> UpdateAttribute (where UpdateAttribute has success relationship terminated) On gui of one of the remaining connected nodes, reconnect disconnected node. It will reconnect successfully and all node graphs will be blank. Cluster docs and/or logs should capture this situation better. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834034#comment-17834034
]
Nissim Shiman commented on NIFI-12969:
--
Thank you very much [~markap14] [~joewitt] [~pgyori] for looking at this and
the resolution.
I did some testing and verified the issue no longer occurs.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 2.0.0-M3, 1.26.0
>
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3
[jira] [Comment Edited] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833705#comment-17833705
]
Nissim Shiman edited comment on NIFI-12969 at 4/3/24 6:45 PM:
--
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out
the finer details of this.
This greatly appreciated.
I am looking at a solution like you suggested (i.e. wait a little in
StandardConnection) and after a short period of time, to give up and in the
calling method undo the temp connections (and remove the temp funnel) to try to
keep the graph the way it was before the temp connection(s) were made.
There doesn't appear to be an easy way to predict this situation to avoid
making the temp connections in the first place. I've tried checking for
unacknowleged flowfiles on all a group's connections before making
temp-funnel/temp destinations (for any connection). But even if it checks
out, by the time we get to setting an individual connection's destination,
there could be unacknowledged flowfiles on the connection again.
Another possibility is to make a special case for this situation (i.e. adding
another
[DisconnectionCode
|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster-protocol/src/main/java/org/apache/nifi/cluster/coordination/node/DisconnectionCode.java]
and in general to have a special case where we try to connect again after
cluster reconnection fails for this case. But this will affect a lot of code
beyond this.
I am very much interested in input on other angles to look at with this.
was (Author: nissim shiman):
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out
the finer details of this.
This greatly appreciated.
I am looking at a solution like you suggested (i.e. wait a little in
StandardConnection) and after a short period of time, to give up and in the
calling method undo the temp connections (and remove the temp funnel) to try to
keep the graph the way it was before the temp connection(s) were made.
There doesn't appear to be an easy way to predict this situation to avoid
making the temp connections in the first place. I've tried checking for
unacknowleged flowfiles on all a group's connections before making
temp-funnel/temp destinations (for any connection). But even if it checks
out, by the time we get to setting an individual connection's destination,
there could be unacknowledged flowfiles on the connection again.
I am very much interested in input on other angles to look at with this.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed
[jira] [Comment Edited] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833705#comment-17833705
]
Nissim Shiman edited comment on NIFI-12969 at 4/3/24 6:29 PM:
--
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out
the finer details of this.
This greatly appreciated.
I am looking at a solution like you suggested (i.e. wait a little in
StandardConnection) and after a short period of time, to give up and in the
calling method undo the temp connections (and remove the temp funnel) to try to
keep the graph the way it was before the temp connection(s) were made.
There doesn't appear to be an easy way to predict this situation to avoid
making the temp connections in the first place. I've tried checking for
unacknowleged flowfiles on all a group's connections before making
temp-funnel/temp destinations (for any connection). But even if it checks
out, by the time we get to setting an individual connection's destination,
there could be unacknowledged flowfiles on the connection again.
I am very much interested in input on other angles to look at with this.
was (Author: nissim shiman):
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out
the finer details of this.
This greatly appreciated.
I am looking at a solution like you suggested (i.e. wait a little in
StandardConnection) and after a short period of time, to give up and in the
calling method undo the temp connections (and remove the temp funnel) to try to
keep the graph the way it was before the temp connection(s) were made.
There doesn't appear to be an easy way to predict this situation to avoid
making the temp connections in the first place. I've tried checking for
unacknowleged flowfiles on all a group's connections before making
temp-funnel/temp destinations (for any connection). But even if it checks
out, by the time we get to setting an individual connection's destination,
there could be unacknowledged flowfiles on the connection again.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
>
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833705#comment-17833705
]
Nissim Shiman commented on NIFI-12969:
--
Thank you very much [~pgyori] for the testing, diagrams, logs and bringing out
the finer details of this.
This greatly appreciated.
I am looking at a solution like you suggested (i.e. wait a little in
StandardConnection) and after a short period of time, to give up and in the
calling method undo the temp connections (and remove the temp funnel) to try to
keep the graph the way it was before the temp connection(s) were made.
There doesn't appear to be an easy way to predict this situation to avoid
making the temp connections in the first place. I've tried checking for
unacknowleged flowfiles on all a group's connections before making
temp-funnel/temp destinations (for any connection). But even if it checks
out, by the time we get to setting an individual connection's destination,
there could be unacknowledged flowfiles on the connection again.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Attachments: nifi-app.log, simple_flow.png,
> simple_flow_with_temp-funnel.png
>
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
>
[jira] [Commented] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832242#comment-17832242
]
Nissim Shiman commented on NIFI-12969:
--
[~pvillard] This is an excellent observation as the symptoms are very similar.
I tried this on a 2.0.0-SNAPSHOT that is post the NIFI-12232 fix, but the issue
still remains.
On closer look, it appears the cause of this issue (in StandardConnection.java)
is the next line after the one that caused NIFI-12232, so maybe that one was
masking some occurrences of this this one, but, yes, you are correct they are
in the same area of code.
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
>
[jira] [Assigned] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman reassigned NIFI-12969:
Assignee: Nissim Shiman
> Under heavy load, nifi node unable to rejoin cluster, graph modified with
> temp funnel
> -
>
> Key: NIFI-12969
> URL: https://issues.apache.org/jira/browse/NIFI-12969
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.24.0, 2.0.0-M2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
>
> Under heavy load, if a node leaves the cluster (due to heartbeat time out),
> many times it is unable to rejoin the cluster.
> The nodes' graph will have been modified with a temp-funnel as well.
> Appears to be some sort of [timing
> issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
> # To reproduce, on a nifi cluster of three nodes, set up:
> 2 GenerateFlowFile processors -> PG
> Inside PG:
> inputPort -> UpdateAttribute
> # Keep all defaults except for the following:
> For UpdateAttribute terminate the success relationship
> One of the GenerateFlowFile processors can be disabled,
> the other one should have Run Schedule to be 0 min (this will allow for the
> heavy load)
> # In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
> cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
> nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
> Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
> After a few minutes one will likely not be able to rejoin. The graph for
> that node will have the disabled GenerateFlowFile now pointing to a funnel (a
> temp-funnel) instead of the PG
> Stack trace on that nodes nifi-app.log will look like this: (this is from
> 2.0.0-M2):
> {code:java}
> 2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
> properly handle Reconnection request due to org.apache.nifi.control
> ler.serialization.FlowSynchronizationException: Failed to connect node to
> cluster because local flow controller partially updated. Administrator should
> disconnect node and review flow for corrup
> tion.
> 2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
> o.a.nifi.controller.StandardFlowService Handling reconnection request failed
> due to: org.apache.nifi.controller.serialization.FlowSynchroniza
> tionException: Failed to connect node to cluster because local flow
> controller partially updated. Administrator should disconnect node and review
> flow for corruption.
> org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
> to connect node to cluster because local flow controller partially updated.
> Administrator should disconnect node and
> review flow for corruption.
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
> at
> org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalStateException: Cannot change destination of Connection
> because FlowFiles from this Connection
> are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
> type=INPUT_PORT, name=inputPort, group=innerPG]
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
> at
> org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
> at
> org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
> ... 3 common frames omitted
> Caused by: java.lang.IllegalStateException: Cannot change destination of
> Connection because FlowFiles from this Connection are currently held by
> LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b
> , type=INPUT_PORT, name=inputPort, group=innerPG]
> at
>
[jira] [Updated] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
[
https://issues.apache.org/jira/browse/NIFI-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-12969:
-
Description:
Under heavy load, if a node leaves the cluster (due to heartbeat time out),
many times it is unable to rejoin the cluster.
The nodes' graph will have been modified with a temp-funnel as well.
Appears to be some sort of [timing
issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
# To reproduce, on a nifi cluster of three nodes, set up:
2 GenerateFlowFile processors -> PG
Inside PG:
inputPort -> UpdateAttribute
# Keep all defaults except for the following:
For UpdateAttribute terminate the success relationship
One of the GenerateFlowFile processors can be disabled,
the other one should have Run Schedule to be 0 min (this will allow for the
heavy load)
# In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
After a few minutes one will likely not be able to rejoin. The graph for that
node will have the disabled GenerateFlowFile now pointing to a funnel (a
temp-funnel) instead of the PG
Stack trace on that nodes nifi-app.log will look like this: (this is from
2.0.0-M2):
{code:java}
2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
properly handle Reconnection request due to org.apache.nifi.control
ler.serialization.FlowSynchronizationException: Failed to connect node to
cluster because local flow controller partially updated. Administrator should
disconnect node and review flow for corrup
tion.
2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
o.a.nifi.controller.StandardFlowService Handling reconnection request failed
due to: org.apache.nifi.controller.serialization.FlowSynchroniza
tionException: Failed to connect node to cluster because local flow controller
partially updated. Administrator should disconnect node and review flow for
corruption.
org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
to connect node to cluster because local flow controller partially updated.
Administrator should disconnect node and
review flow for corruption.
at
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
at
org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
at
org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by:
org.apache.nifi.controller.serialization.FlowSynchronizationException:
java.lang.IllegalStateException: Cannot change destination of Connection
because FlowFiles from this Connection
are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
type=INPUT_PORT, name=inputPort, group=innerPG]
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
at
org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
at
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
at
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
... 3 common frames omitted
Caused by: java.lang.IllegalStateException: Cannot change destination of
Connection because FlowFiles from this Connection are currently held by
LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b
, type=INPUT_PORT, name=inputPort, group=innerPG]
at
org.apache.nifi.connectable.StandardConnection.setDestination(StandardConnection.java:299)
at
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.updateConnectionDestinations(StandardVersionedComponentSynchronizer.java:705)
at
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:423)
at
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:248)
at
org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:638)
at
[jira] [Created] (NIFI-12969) Under heavy load, nifi node unable to rejoin cluster, graph modified with temp funnel
Nissim Shiman created NIFI-12969:
Summary: Under heavy load, nifi node unable to rejoin cluster,
graph modified with temp funnel
Key: NIFI-12969
URL: https://issues.apache.org/jira/browse/NIFI-12969
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.0.0-M2, 1.24.0
Reporter: Nissim Shiman
Under heavy load, if a node leaves the cluster (due to heartbeat time out),
many times it is unable to rejoin the cluster.
The nodes' graph will been modified with a temp-funnel as well.
Appears to be some sort of [timing
issue|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/connectable/StandardConnection.java#L298]
# To reproduce, on a nifi cluster of three nodes, set up:
2 GenerateFlowFile processors -> PG
Inside PG:
inputPort -> UpdateAttribute
# Keep all defaults except for the following:
For UpdateAttribute terminate the success relationship
One of the GenerateFlowFile processors can be disabled,
the other one should have Run Schedule to be 0 min (this will allow for the
heavy load)
# In nifi.properties (on all 3 nodes) to allow for nodes to fall out of the
cluster, set: nifi.cluster.protocol.heartbeat.interval=2 sec (default is 5)
nifi.cluster.protocol.heartbeat.missable.max=1 (default is 8)
Restart nifi. Start flow. The nodes will quickly fall out and rejoin cluster.
After a few minutes one will likely not be able to rejoin. The graph for that
node will have the disabled GenerateFlowFile now pointing to a funnel (a
temp-funnel) instead of the PG
Stack trace on that nodes nifi-app.log will look like this: (this is from
2.0.0-M2):
{code:java}
2024-03-28 13:55:19,395 INFO [Reconnect to Cluster]
o.a.nifi.controller.StandardFlowService Node disconnected due to Failed to
properly handle Reconnection request due to org.apache.nifi.control
ler.serialization.FlowSynchronizationException: Failed to connect node to
cluster because local flow controller partially updated. Administrator should
disconnect node and review flow for corrup
tion.
2024-03-28 13:55:19,395 ERROR [Reconnect to Cluster]
o.a.nifi.controller.StandardFlowService Handling reconnection request failed
due to: org.apache.nifi.controller.serialization.FlowSynchroniza
tionException: Failed to connect node to cluster because local flow controller
partially updated. Administrator should disconnect node and review flow for
corruption.
org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed
to connect node to cluster because local flow controller partially updated.
Administrator should disconnect node and
review flow for corruption.
at
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:985)
at
org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:655)
at
org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:384)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by:
org.apache.nifi.controller.serialization.FlowSynchronizationException:
java.lang.IllegalStateException: Cannot change destination of Connection
because FlowFiles from this Connection
are currently held by LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b,
type=INPUT_PORT, name=inputPort, group=innerPG]
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:472)
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:223)
at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1740)
at
org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
at
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
at
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:954)
... 3 common frames omitted
Caused by: java.lang.IllegalStateException: Cannot change destination of
Connection because FlowFiles from this Connection are currently held by
LocalPort[id=99213c00-78ca-4848-112f-5454cc20656b
, type=INPUT_PORT, name=inputPort, group=innerPG]
at
org.apache.nifi.connectable.StandardConnection.setDestination(StandardConnection.java:299)
at
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.updateConnectionDestinations(StandardVersionedComponentSynchronizer.java:705)
at
org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:423)
at
[jira] [Created] (NIFI-12878) Enable flow history for process groups' Enable/Disable all controller services option(s) for controller services
Nissim Shiman created NIFI-12878: Summary: Enable flow history for process groups' Enable/Disable all controller services option(s) for controller services Key: NIFI-12878 URL: https://issues.apache.org/jira/browse/NIFI-12878 Project: Apache NiFi Issue Type: Improvement Reporter: Nissim Shiman When a process group's "Enable all controller services" option (or "Disable all controller services" option) is chosen, all controller services in that PG are enabled (or disabled). Flow configuration history captures the PG that was enabled/disabled, but the specific controller services that were enabled/disabled are not captured. It would be useful if the specific controller services that were enabled/disabled could be captured here as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12878) Enable flow history for process groups' Enable/Disable all controller services option(s) for controller services
[ https://issues.apache.org/jira/browse/NIFI-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-12878: Assignee: Nissim Shiman > Enable flow history for process groups' Enable/Disable all controller > services option(s) for controller services > > > Key: NIFI-12878 > URL: https://issues.apache.org/jira/browse/NIFI-12878 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > When a process group's "Enable all controller services" option (or "Disable > all controller services" option) is chosen, all controller services in that > PG are enabled (or disabled). > Flow configuration history captures the PG that was enabled/disabled, but the > specific controller services that were enabled/disabled are not captured. > It would be useful if the specific controller services that were > enabled/disabled could be captured here as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11871) Controller Services gui does not automatically update Last Updated time
[ https://issues.apache.org/jira/browse/NIFI-11871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11871: - Description: The Controller Service gui (right click -> Configure -> Controller Services) has a "Last Updated" time in bottom left hand corner. This time will stay the same, eventhough the refresh circle flashes every 30 seconds (or whatever the ui autorefresh interval is set to) to the right of this time. Note: This can still be refreshed manually (and "Last Updated" time will be set to current time). Update 2/29/24: This work should be done on the new 2.0 gui was: The Controller Service gui (right click -> Configure -> Controller Services) has a "Last Updated" time in bottom left hand corner. This time will stay the same, eventhough the refresh circle flashes every 30 seconds (or whatever the ui autorefresh interval is set to) to the right of this time. Note: This can still be refreshed manually (and "Last Updated" time will be set to current time). > Controller Services gui does not automatically update Last Updated time > --- > > Key: NIFI-11871 > URL: https://issues.apache.org/jira/browse/NIFI-11871 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > The Controller Service gui (right click -> Configure -> Controller Services) > has a "Last Updated" time in bottom left hand corner. > This time will stay the same, eventhough the refresh circle flashes every 30 > seconds (or whatever the ui autorefresh interval is set to) to the right of > this time. > Note: This can still be refreshed manually (and "Last Updated" time will be > set to current time). > Update 2/29/24: This work should be done on the new 2.0 gui -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9579) swagger-maven-plugin depends on log4j. Use newer one (when made available)
[
https://issues.apache.org/jira/browse/NIFI-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813813#comment-17813813
]
Nissim Shiman commented on NIFI-9579:
-
Issue is resolved as of NIFI-11703 (Thank you [~exceptionfactory] ) for apache
nifi 2.0 branch
Not sure if this solution can work on 1.x branch as solution uses
[swagger-maven-plugin-jakarta|https://github.com/apache/nifi/blob/rel/nifi-2.0.0-M2/nifi-registry/nifi-registry-core/nifi-registry-web-api/pom.xml#L91]
and when moving the changes from NIFI-11703 and compiling with java 8, am
seeing
{code}
Execution default of goal
io.swagger.core.v3:swagger-maven-plugin-jakarta:2.2.19:resolve failed: An API
incompatibility was encountered while executing
io.swagger.core.v3:swagger-maven-plugin-jakarta:2.2.19:resolve:
java.lang.UnsupportedClassVersionError: jakarta/ws/rs/Path has been compiled by
a more recent version of the Java Runtime (class file version 55.0), this
version of the Java Runtime only recognizes class file versions up to 52.0
{code}
(This is not seen when compiling on java 11)
> swagger-maven-plugin depends on log4j. Use newer one (when made available)
> ---
>
> Key: NIFI-9579
> URL: https://issues.apache.org/jira/browse/NIFI-9579
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.15.2
>Reporter: Nissim Shiman
>Priority: Major
>
> Note: This only affects nifi dev environments - not the apache nifi product.
> At the moment building apache nifi will result in log4j-1.2.16.jar in one's
> maven local repo.
> This is due to swagger-maven-plugins which pulls these in when used [1] [2]
> swagger-maven-plugin is know to have this dependency but it is unclear how
> soon this will be acted on
> [3] . When swagger-maven-plugin has this resolved, update to new version.
>
> [1]
> [https://github.com/apache/nifi/blob/rel/nifi-1.15.2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/pom.xml#L60-L62|https://github.com/apache/nifi/blob/rel/nifi-1.15.2/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/pom.xml#L61-L62]
> [2]
> [https://github.com/apache/nifi/blob/rel/nifi-1.15.2/nifi-registry/nifi-registry-core/nifi-registry-web-api/pom.xml#L55-L57]
> [3] [https://github.com/kongchen/swagger-maven-plugin/issues/875]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12678) Link from Reporting Task to referenced Controller Service not fully working
Nissim Shiman created NIFI-12678: Summary: Link from Reporting Task to referenced Controller Service not fully working Key: NIFI-12678 URL: https://issues.apache.org/jira/browse/NIFI-12678 Project: Apache NiFi Issue Type: Bug Reporter: Nissim Shiman If a reporting task is referencing a controller service, clicking on the arrow leading to the controller service will bring user to the Management Controller Services tab, but will point to the top controller service on the list. So, for example, if SiteToSiteProvenanceReportingTask has SSL Context Service set, clicking on the arrow to the right of the value will take us to the Management Controller Services tab, but will point to the top controller service on the list. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12387) Flow Configuration History can record a Comments change for Controller Service when no changes have been made
[
https://issues.apache.org/jira/browse/NIFI-12387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-12387:
-
Affects Version/s: 2.0.0-M1
Labels: backport-needed (was: )
Status: Patch Available (was: Open)
> Flow Configuration History can record a Comments change for Controller
> Service when no changes have been made
> -
>
> Key: NIFI-12387
> URL: https://issues.apache.org/jira/browse/NIFI-12387
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.23.2, 1.16.3, 2.0.0-M1
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Labels: backport-needed
> Time Spent: 10m
> Remaining Estimate: 0h
>
> To recreate, create a new controller service then open configuration gui for
> it and click Apply button (without having made any changes to default
> settings).
> Flow Configuration History will note that a Configure operation has occurred
> and the Comments have been changed from _No value set_ to an {_}Empty string
> se{_}t.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12598) Enable flow history for start/stop of process group components
Nissim Shiman created NIFI-12598: Summary: Enable flow history for start/stop of process group components Key: NIFI-12598 URL: https://issues.apache.org/jira/browse/NIFI-12598 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.24.0 Reporter: Nissim Shiman Assignee: Nissim Shiman When a processor group is started/stopped it is captured in flow configuration history, but its internal components (i.e. processors, input ports, output ports) are not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12387) Flow Configuration History can record a Comments change for Controller Service when no changes have been made
[
https://issues.apache.org/jira/browse/NIFI-12387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman reassigned NIFI-12387:
Assignee: Nissim Shiman
> Flow Configuration History can record a Comments change for Controller
> Service when no changes have been made
> -
>
> Key: NIFI-12387
> URL: https://issues.apache.org/jira/browse/NIFI-12387
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.16.3, 1.23.2
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
>
> To recreate, create a new controller service then open configuration gui for
> it and click Apply button (without having made any changes to default
> settings).
> Flow Configuration History will note that a Configure operation has occurred
> and the Comments have been changed from _No value set_ to an {_}Empty string
> se{_}t.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11389) Controller Services's link to referencing Controller Services components not always working
[ https://issues.apache.org/jira/browse/NIFI-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11389: - Labels: backport-needed (was: ) > Controller Services's link to referencing Controller Services components not > always working > --- > > Key: NIFI-11389 > URL: https://issues.apache.org/jira/browse/NIFI-11389 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 2.0.0-M1, 1.20.0, 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Labels: backport-needed > Time Spent: 10m > Remaining Estimate: 0h > > Controller Services' Setting tab has a _Referencing Components_ section on > the right side. > There are hyperlinks to the Processors and/or Controller Services which > reference this controller service. These links do not always work when > referencing controller services. > To reproduce: > Create StandardRestrictedSSLContextService on top level nifi graph (i.e. > right click top level grid -> Configure -> Controller Services) > Create Process Group > Inside Process Group create DistributedMapCacheServer > and point _SSL Context Service_ to StandardRestrictedSSLContextService just > created > Go to StandardRestrictedSSLContextService's Setting's tab and verify > DistributedMapCacheServer appears in the _Referencing Components_ section. > Clicking on the link to DistributedMapCacheServer will not take us to that > controller service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11570) Processors have difficulty restoring dynamic properties when leaving ghost mode
[ https://issues.apache.org/jira/browse/NIFI-11570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799130#comment-17799130 ] Nissim Shiman commented on NIFI-11570: -- Circling back to this... This will involve changes to flow.json so I am hoping to have more people comment before comitting to an approach. Thank you very much [~bbende] for your approach (linked above) of storing the set of sensitiveDynamicPropertyNames in flow.json. I ran this problem by a senior engineer, who suggested that using the (currently existing, but empty) propertyDescriptor object in flow.json is a good approach as well. This may end of touching many files, so I would very much appreciate more feedback on this (more approaches or comments to the above two ideas) > Processors have difficulty restoring dynamic properties when leaving ghost > mode > --- > > Key: NIFI-11570 > URL: https://issues.apache.org/jira/browse/NIFI-11570 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > When a processor with non sensitive dynamic properties is started in ghost > mode due to its underlying nar being missing, it will have a difficult time > being restored even when its nar has been replaced. > The dynamic properties will be identified as being sensitive and not able to > be accessed. > So for example, if a dynamic property added to GenerateFlowFile and > nifi-standard-nar is removed from lib and nifi is restarted, GenerateFlowFile > will come up in ghosted mode (as expected). > When nifi-standard-nar is replaced and nifi is restarted, GenerateFlowFile > will be restored, but its dynamic property will not be accessable. > This was originally filed as a second part of NIFI-11109, but after looking > more into the underlying causes/symptoms, realized that these are two > different issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11389) Controller Services's link to referencing Controller Services components not always working
[ https://issues.apache.org/jira/browse/NIFI-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11389: - Affects Version/s: 2.0.0-M1 Status: Patch Available (was: Open) > Controller Services's link to referencing Controller Services components not > always working > --- > > Key: NIFI-11389 > URL: https://issues.apache.org/jira/browse/NIFI-11389 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.19.1, 1.20.0, 2.0.0-M1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Controller Services' Setting tab has a _Referencing Components_ section on > the right side. > There are hyperlinks to the Processors and/or Controller Services which > reference this controller service. These links do not always work when > referencing controller services. > To reproduce: > Create StandardRestrictedSSLContextService on top level nifi graph (i.e. > right click top level grid -> Configure -> Controller Services) > Create Process Group > Inside Process Group create DistributedMapCacheServer > and point _SSL Context Service_ to StandardRestrictedSSLContextService just > created > Go to StandardRestrictedSSLContextService's Setting's tab and verify > DistributedMapCacheServer appears in the _Referencing Components_ section. > Clicking on the link to DistributedMapCacheServer will not take us to that > controller service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12387) Flow Configuration History can record a Comments change for Controller Service when no changes have been made
[
https://issues.apache.org/jira/browse/NIFI-12387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-12387:
-
Affects Version/s: 1.23.2
1.16.3
> Flow Configuration History can record a Comments change for Controller
> Service when no changes have been made
> -
>
> Key: NIFI-12387
> URL: https://issues.apache.org/jira/browse/NIFI-12387
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.16.3, 1.23.2
>Reporter: Nissim Shiman
>Priority: Major
>
> To recreate, create a new controller service then open configuration gui for
> it and click Apply button (without having made any changes to default
> settings).
> Flow Configuration History will note that a Configure operation has occurred
> and the Comments have been changed from _No value set_ to an {_}Empty string
> se{_}t.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12388) Process Group log file suffix can become out of sync with its name when PG was copied from a version controlled PG
Nissim Shiman created NIFI-12388: Summary: Process Group log file suffix can become out of sync with its name when PG was copied from a version controlled PG Key: NIFI-12388 URL: https://issues.apache.org/jira/browse/NIFI-12388 Project: Apache NiFi Issue Type: Bug Reporter: Nissim Shiman To recreate, make a log suffix for a versioned controlled PG. (i.e. right click on PG -> Configure -> General -> Log File Suffix) Copy PG Notice the name is same as the original PG but the Log File Suffix will be have the words "Copy of" preceding it. This does not occur with a copy of a non-versioned controlled PG (i.e. "Copy of" is pre-pended to both the new PG name and Log File Suffix so it is more in sync) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12387) Flow Configuration History can record a Comments change for Controller Service when no changes have been made
Nissim Shiman created NIFI-12387:
Summary: Flow Configuration History can record a Comments change
for Controller Service when no changes have been made
Key: NIFI-12387
URL: https://issues.apache.org/jira/browse/NIFI-12387
Project: Apache NiFi
Issue Type: Bug
Reporter: Nissim Shiman
To recreate, create a new controller service then open configuration gui for it
and click Apply button (without having made any changes to default settings).
Flow Configuration History will note that a Configure operation has occurred
and the Comments have been changed from _No value set_ to an {_}Empty string
se{_}t.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11463) Add option for IdentifyMimeType custom mime definitions to be in addition to the default tika definitions
[ https://issues.apache.org/jira/browse/NIFI-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784082#comment-17784082 ] Nissim Shiman commented on NIFI-11463: -- Thank you [~exceptionfactory] for direction in the PR for this and [~markap14] for the really nice work for migrateProperties() (NIFI-12139 , NIFI-12301). Both were incorporated in this, and made the code more graceful. It is very much appreciated. > Add option for IdentifyMimeType custom mime definitions to be in addition to > the default tika definitions > - > > Key: NIFI-11463 > URL: https://issues.apache.org/jira/browse/NIFI-11463 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Fix For: 2.0.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > IdentifyMimeType has options for users to configure custom mime types > definitions. > These user defined definitions override the default tika mime type > definitions. > Add option to allow custom mime type definitions to be in addition to the > default tika mime type definitions. > Users will be able to decide if custom definitions should be used instead of > or in addition to default tika definitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12315) Difficulty importing PG from nifi registry
[ https://issues.apache.org/jira/browse/NIFI-12315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-12315: - Status: Patch Available (was: In Progress) > Difficulty importing PG from nifi registry > -- > > Key: NIFI-12315 > URL: https://issues.apache.org/jira/browse/NIFI-12315 > Project: Apache NiFi > Issue Type: Bug >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > It appears that it is no longer possible to import a PG from nifi registry > via the nifi ui. > (i.e. Add Process Group -> Import from Registry > no longer works) > This is occurring on main/2.x build from this morning (11/3/23), not in any > released tags. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12315) Difficulty importing PG from nifi registry
[ https://issues.apache.org/jira/browse/NIFI-12315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-12315: Assignee: Nissim Shiman > Difficulty importing PG from nifi registry > -- > > Key: NIFI-12315 > URL: https://issues.apache.org/jira/browse/NIFI-12315 > Project: Apache NiFi > Issue Type: Bug >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > It appears that it is no longer possible to import a PG from nifi registry > via the nifi ui. > (i.e. Add Process Group -> Import from Registry > no longer works) > This is occurring on main/2.x build from this morning (11/3/23), not in any > released tags. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12315) Difficulty importing PG from nifi registry
Nissim Shiman created NIFI-12315: Summary: Difficulty importing PG from nifi registry Key: NIFI-12315 URL: https://issues.apache.org/jira/browse/NIFI-12315 Project: Apache NiFi Issue Type: Bug Reporter: Nissim Shiman It appears that it is no longer possible to import a PG from nifi registry via the nifi ui. (i.e. Add Process Group -> Import from Registry no longer works) This is occurring on main/2.x build from this morning (11/3/23), not in any released tags. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11782) NPE when adding Snippet with label into Process Group
[
https://issues.apache.org/jira/browse/NIFI-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-11782:
-
Status: Patch Available (was: In Progress)
> NPE when adding Snippet with label into Process Group
> -
>
> Key: NIFI-11782
> URL: https://issues.apache.org/jira/browse/NIFI-11782
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.22.0, 1.21.0
>Reporter: Pierre Villard
>Assignee: Nissim Shiman
>Priority: Major
> Fix For: 1.latest, 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When adding to a Process Group a snippet that contains a label
> {code:java}
> 2023-07-06 15:35:47,575 ERROR [NiFi Web Server-222]
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> java.lang.NullPointerException: Cannot invoke
> "org.apache.nifi.web.dao.LabelDAO.getLabel(String)" because "this.labelDAO"
> is null. Returning Internal Server Error response.
> java.lang.NullPointerException: Cannot invoke
> "org.apache.nifi.web.dao.LabelDAO.getLabel(String)" because "this.labelDAO"
> is null
> at
> org.apache.nifi.audit.SnippetAuditor.updateSnippetAdvice(SnippetAuditor.java:336)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> {code}
> This seems to be caused by NIFI-11134.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (NIFI-11084) Character/text data "mis-identified" by IdentifyMimeType processor
[
https://issues.apache.org/jira/browse/NIFI-11084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768114#comment-17768114
]
Nissim Shiman edited comment on NIFI-11084 at 10/3/23 7:10 PM:
---
Once NIFI-11463 is complete, a custom rule can be made for this [1], which will
hit before reaching the tika rules. This would elevate csv rule processing
above image/x-portable-graymap (Thank you [~dstieg1] for pointing this out to
me].
[1] Config Strategy would be set to "Add"
and Config Body would be:
{code:java}
{code}
Update: 10/3/23: During code review for NIFI-11463, an alternative
implementation was discussed and decided upon where the above workaround would
no longer be possible.
Apologies for raising premature hopes about this.
was (Author: nissim shiman):
Once NIFI-11463 is complete, a custom rule can be made for this [1], which will
hit before reaching the tika rules. This would elevate csv rule processing
above image/x-portable-graymap (Thank you [~dstieg1] for pointing this out to
me].
[1] Config Strategy would be set to "Add"
and Config Body would be:
{code:java}
{code}
> Character/text data "mis-identified" by IdentifyMimeType processor
> --
>
> Key: NIFI-11084
> URL: https://issues.apache.org/jira/browse/NIFI-11084
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.15.2, 1.19.1
> Environment: Windows Server 2019, Java 11.0.17
>Reporter: Mark Ward
>Priority: Minor
> Attachments: mime_type_mis-id_file.csv
>
>
> When *IdentifyMimeType* is presented with a text file with a `.csv` extension
> and the first two characters of the content as `P2`, the processor
> mis-identifies the mime.extension as `pgm` and mime.type as
> `image/x-portable-graymap`.
> The processor's *Use Filename In Detection* property is set to `true`.
> An example file is attached and the following flow and be used to reproduce:
> *GetFile* > *IdentifyMimeType* where the outputted flowfile's attributes can
> be inspected.
> This has been tested on NiFi versions `1.15.2` and `1.19.1` both running on a
> Window's Server 2019 instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-11782) NPE when adding Snippet with label into Process Group
[
https://issues.apache.org/jira/browse/NIFI-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman reassigned NIFI-11782:
Assignee: Nissim Shiman
> NPE when adding Snippet with label into Process Group
> -
>
> Key: NIFI-11782
> URL: https://issues.apache.org/jira/browse/NIFI-11782
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.21.0, 1.22.0
>Reporter: Pierre Villard
>Assignee: Nissim Shiman
>Priority: Major
> Fix For: 1.latest, 2.latest
>
>
> When adding to a Process Group a snippet that contains a label
> {code:java}
> 2023-07-06 15:35:47,575 ERROR [NiFi Web Server-222]
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> java.lang.NullPointerException: Cannot invoke
> "org.apache.nifi.web.dao.LabelDAO.getLabel(String)" because "this.labelDAO"
> is null. Returning Internal Server Error response.
> java.lang.NullPointerException: Cannot invoke
> "org.apache.nifi.web.dao.LabelDAO.getLabel(String)" because "this.labelDAO"
> is null
> at
> org.apache.nifi.audit.SnippetAuditor.updateSnippetAdvice(SnippetAuditor.java:336)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> {code}
> This seems to be caused by NIFI-11134.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (NIFI-11084) Character/text data "mis-identified" by IdentifyMimeType processor
[
https://issues.apache.org/jira/browse/NIFI-11084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768114#comment-17768114
]
Nissim Shiman edited comment on NIFI-11084 at 9/29/23 12:36 PM:
Once NIFI-11463 is complete, a custom rule can be made for this [1], which will
hit before reaching the tika rules. This would elevate csv rule processing
above image/x-portable-graymap (Thank you [~dstieg1] for pointing this out to
me].
[1] Config Strategy would be set to "Add"
and Config Body would be:
{code:java}
{code}
was (Author: nissim shiman):
Once NIFI-11463 is complete, a custom rule can be made for this, which will hit
before reaching the tika rules. (Thank you [~dstieg1] for pointing this out to
me].
> Character/text data "mis-identified" by IdentifyMimeType processor
> --
>
> Key: NIFI-11084
> URL: https://issues.apache.org/jira/browse/NIFI-11084
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.15.2, 1.19.1
> Environment: Windows Server 2019, Java 11.0.17
>Reporter: Mark Ward
>Priority: Minor
> Attachments: mime_type_mis-id_file.csv
>
>
> When *IdentifyMimeType* is presented with a text file with a `.csv` extension
> and the first two characters of the content as `P2`, the processor
> mis-identifies the mime.extension as `pgm` and mime.type as
> `image/x-portable-graymap`.
> The processor's *Use Filename In Detection* property is set to `true`.
> An example file is attached and the following flow and be used to reproduce:
> *GetFile* > *IdentifyMimeType* where the outputted flowfile's attributes can
> be inspected.
> This has been tested on NiFi versions `1.15.2` and `1.19.1` both running on a
> Window's Server 2019 instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-11871) Controller Services gui does not automatically update Last Updated time
[ https://issues.apache.org/jira/browse/NIFI-11871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-11871: Assignee: Nissim Shiman > Controller Services gui does not automatically update Last Updated time > --- > > Key: NIFI-11871 > URL: https://issues.apache.org/jira/browse/NIFI-11871 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > The Controller Service gui (right click -> Configure -> Controller Services) > has a "Last Updated" time in bottom left hand corner. > This time will stay the same, eventhough the refresh circle flashes every 30 > seconds (or whatever the ui autorefresh interval is set to) to the right of > this time. > Note: This can still be refreshed manually (and "Last Updated" time will be > set to current time). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11463) Add option for IdentifyMimeType custom mime definitions to be in addition to the default tika definitions
[ https://issues.apache.org/jira/browse/NIFI-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11463: - Status: Patch Available (was: Open) > Add option for IdentifyMimeType custom mime definitions to be in addition to > the default tika definitions > - > > Key: NIFI-11463 > URL: https://issues.apache.org/jira/browse/NIFI-11463 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > IdentifyMimeType has options for users to configure custom mime types > definitions. > These user defined definitions override the default tika mime type > definitions. > Add option to allow custom mime type definitions to be in addition to the > default tika mime type definitions. > Users will be able to decide if custom definitions should be used instead of > or in addition to default tika definitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11084) Character/text data "mis-identified" by IdentifyMimeType processor
[ https://issues.apache.org/jira/browse/NIFI-11084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768114#comment-17768114 ] Nissim Shiman commented on NIFI-11084: -- Once NIFI-11463 is complete, a custom rule can be made for this, which will hit before reaching the tika rules. (Thank you [~dstieg1] for pointing this out to me]. > Character/text data "mis-identified" by IdentifyMimeType processor > -- > > Key: NIFI-11084 > URL: https://issues.apache.org/jira/browse/NIFI-11084 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.15.2, 1.19.1 > Environment: Windows Server 2019, Java 11.0.17 >Reporter: Mark Ward >Priority: Minor > Attachments: mime_type_mis-id_file.csv > > > When *IdentifyMimeType* is presented with a text file with a `.csv` extension > and the first two characters of the content as `P2`, the processor > mis-identifies the mime.extension as `pgm` and mime.type as > `image/x-portable-graymap`. > The processor's *Use Filename In Detection* property is set to `true`. > An example file is attached and the following flow and be used to reproduce: > *GetFile* > *IdentifyMimeType* where the outputted flowfile's attributes can > be inspected. > This has been tested on NiFi versions `1.15.2` and `1.19.1` both running on a > Window's Server 2019 instance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11872) Controller Services can appear to be stuck in Validating state immediately after being disabled
Nissim Shiman created NIFI-11872: Summary: Controller Services can appear to be stuck in Validating state immediately after being disabled Key: NIFI-11872 URL: https://issues.apache.org/jira/browse/NIFI-11872 Project: Apache NiFi Issue Type: Bug Reporter: Nissim Shiman A controller service will go into Validating state immediately after being disabled and controller service gui will not automatically clear this state. This occurs to me about 50% of the time when disabling a controller service. State will remain "Validating" with the spinning wheel next to it. Manually refreshing the controller services gui clears this issue and will show the controller service as Disabled -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11872) Controller Services can appear to be stuck in Validating state immediately after being disabled
[ https://issues.apache.org/jira/browse/NIFI-11872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11872: - Affects Version/s: 1.20.0 > Controller Services can appear to be stuck in Validating state immediately > after being disabled > --- > > Key: NIFI-11872 > URL: https://issues.apache.org/jira/browse/NIFI-11872 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Nissim Shiman >Priority: Minor > > A controller service will go into Validating state immediately after being > disabled and controller service gui will not automatically clear this state. > This occurs to me about 50% of the time when disabling a controller service. > State will remain "Validating" with the spinning wheel next to it. > Manually refreshing the controller services gui clears this issue and will > show the controller service as Disabled -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11871) Controller Services gui does not automatically update Last Updated time
Nissim Shiman created NIFI-11871: Summary: Controller Services gui does not automatically update Last Updated time Key: NIFI-11871 URL: https://issues.apache.org/jira/browse/NIFI-11871 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.20.0 Reporter: Nissim Shiman The Controller Service gui (right click -> Configure -> Controller Services) has a "Last Updated" time in bottom left hand corner. This time will stay the same, eventhough the refresh circle flashes every 30 seconds (or whatever the ui autorefresh interval is set to) to the right of this time. Note: This can still be refreshed manually (and "Last Updated" time will be set to current time). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11570) Processors have difficulty restoring dynamic properties when leaving ghost mode
[ https://issues.apache.org/jira/browse/NIFI-11570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739212#comment-17739212 ] Nissim Shiman commented on NIFI-11570: -- Discussion here: https://lists.apache.org/thread/68yyx9x0ox34mcg82p9vrdgzlov7dlvj > Processors have difficulty restoring dynamic properties when leaving ghost > mode > --- > > Key: NIFI-11570 > URL: https://issues.apache.org/jira/browse/NIFI-11570 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > When a processor with non sensitive dynamic properties is started in ghost > mode due to its underlying nar being missing, it will have a difficult time > being restored even when its nar has been replaced. > The dynamic properties will be identified as being sensitive and not able to > be accessed. > So for example, if a dynamic property added to GenerateFlowFile and > nifi-standard-nar is removed from lib and nifi is restarted, GenerateFlowFile > will come up in ghosted mode (as expected). > When nifi-standard-nar is replaced and nifi is restarted, GenerateFlowFile > will be restored, but its dynamic property will not be accessable. > This was originally filed as a second part of NIFI-11109, but after looking > more into the underlying causes/symptoms, realized that these are two > different issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11661) MonitorMemory unable to run as cron
Nissim Shiman created NIFI-11661:
Summary: MonitorMemory unable to run as cron
Key: NIFI-11661
URL: https://issues.apache.org/jira/browse/NIFI-11661
Project: Apache NiFi
Issue Type: Bug
Reporter: Nissim Shiman
The MonitorMemory reporting task is unable to run when using CRON driven
Scheduling Strategy.
It will continue trying to start every 30 seconds.
nifi-app.log shows:
{code:java}
2023-06-07 18:34:39,456 ERROR [Timer-Driven Process Thread-8]
o.apache.nifi.controller.MonitorMemory
MonitorMemory[id=971e5e45-0188-1000-b07f-ba4fcfd3cc65] Failed to invoke
@OnScheduled method due to {}
java.lang.NullPointerException: null
at
org.apache.nifi.controller.MonitorMemory.onConfigured(MonitorMemory.java:174)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
at
org.apache.nifi.controller.scheduling.StandardProcessScheduler$2.run(StandardProcessScheduler.java:240)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-07 18:34:39,457 ERROR [Timer-Driven Process Thread-8]
o.a.n.c.s.StandardProcessScheduler Failed to invoke the On-Scheduled Lifecycle
methods of MonitorMemory[id=971e5e45-0188-1000-b07f-ba4fcfd3cc65] due to
java.lang.reflect.InvocationTargetException; administratively yielding this
ReportingTask and will attempt to schedule it again after 30 sec
java.lang.reflect.InvocationTargetException: null
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
at
org.apache.nifi.controller.scheduling.StandardProcessScheduler$2.run(StandardProcessScheduler.java:240)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException: null
at
org.apache.nifi.controller.MonitorMemory.onConfigured(MonitorMemory.java:174)
... 16 common frames omitted
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-11660) jffi.so created in nifi root when adding ScriptedLookupService
Nissim Shiman created NIFI-11660: Summary: jffi.so created in nifi root when adding ScriptedLookupService Key: NIFI-11660 URL: https://issues.apache.org/jira/browse/NIFI-11660 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.21.0 Reporter: Nissim Shiman When ScriptedLookupService is added as a controller service, a file, jffi<19 or 20 digits>.so, is found in the nifi root directory. It appears to be removed on nifi shutdown and is regenerated on nifi startup. This was not occuring in nifi 1.20. The sha256/512 checksum is the same each time. It doesn't appear to be hurting anything, but seems to be unusual behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-11109) registry client class name modified in flow.json/xml when missing nifi-flow-registry-client-nar
[ https://issues.apache.org/jira/browse/NIFI-11109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725892#comment-17725892 ] Nissim Shiman edited comment on NIFI-11109 at 5/24/23 5:34 PM: --- PR: https://github.com/apache/nifi/pull/7273 was (Author: nissim shiman): PR: https://github.com/apache/nifi/pull/7107 > registry client class name modified in flow.json/xml when missing > nifi-flow-registry-client-nar > --- > > Key: NIFI-11109 > URL: https://issues.apache.org/jira/browse/NIFI-11109 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Fix For: 2.0.0, 1.22.0 > > > If nifi is set to use a registry client and nifi-flow-registry-client-nar is > removed from lib, the next nifi restart will result in the registry's class > name (in flow.xml.gz/flow.json.gz) to be modified from > org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient > to > NifiRegistryFlowRegistryClient. > The url property will also be encrypted. > When the nifi-flow-registry-client-nar is returned to lib, and nifi is > restarted, these changes remain and registry is unreachable using this > registry client. > Update: The remainder of this ticket (that has had its text struck through) > has been made into its own ticket (NIFI-11570) as the underlying issue has > been found to be different. > -Also, if the nar removed was nifi-standard-services-api-nar, then besides > the above behavior, processors under version control via this registry client > may also have their dynamic properties encrypted. These properties remain > encrypted even after nifi-standard-services-api-nar is returned to lib and > nifi is restarted.- > -This is seen with a dynamic property added to GenerateFlowFile (when > GenericFlowFile is part of a PG under registry version control).- > -These are edge cases as admins should be very careful about removing nars > from lib, but it would be good if protections were added to protect > flow.xml/json from modifications in these situations.- -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11109) registry client class name modified in flow.json/xml when missing nifi-flow-registry-client-nar
[ https://issues.apache.org/jira/browse/NIFI-11109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725892#comment-17725892 ] Nissim Shiman commented on NIFI-11109: -- PR: https://github.com/apache/nifi/pull/7107 > registry client class name modified in flow.json/xml when missing > nifi-flow-registry-client-nar > --- > > Key: NIFI-11109 > URL: https://issues.apache.org/jira/browse/NIFI-11109 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > If nifi is set to use a registry client and nifi-flow-registry-client-nar is > removed from lib, the next nifi restart will result in the registry's class > name (in flow.xml.gz/flow.json.gz) to be modified from > org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient > to > NifiRegistryFlowRegistryClient. > The url property will also be encrypted. > When the nifi-flow-registry-client-nar is returned to lib, and nifi is > restarted, these changes remain and registry is unreachable using this > registry client. > Update: The remainder of this ticket (that has had its text struck through) > has been made into its own ticket (NIFI-11570) as the underlying issue has > been found to be different. > -Also, if the nar removed was nifi-standard-services-api-nar, then besides > the above behavior, processors under version control via this registry client > may also have their dynamic properties encrypted. These properties remain > encrypted even after nifi-standard-services-api-nar is returned to lib and > nifi is restarted.- > -This is seen with a dynamic property added to GenerateFlowFile (when > GenericFlowFile is part of a PG under registry version control).- > -These are edge cases as admins should be very careful about removing nars > from lib, but it would be good if protections were added to protect > flow.xml/json from modifications in these situations.- -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11109) registry client class name modified in flow.json/xml when missing nifi-flow-registry-client-nar
[ https://issues.apache.org/jira/browse/NIFI-11109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11109: - Summary: registry client class name modified in flow.json/xml when missing nifi-flow-registry-client-nar (was: flow.json/xml modified when using registry client while missing nifi-flow-registry-client-nar) > registry client class name modified in flow.json/xml when missing > nifi-flow-registry-client-nar > --- > > Key: NIFI-11109 > URL: https://issues.apache.org/jira/browse/NIFI-11109 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > If nifi is set to use a registry client and nifi-flow-registry-client-nar is > removed from lib, the next nifi restart will result in the registry's class > name (in flow.xml.gz/flow.json.gz) to be modified from > org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient > to > NifiRegistryFlowRegistryClient. > The url property will also be encrypted. > When the nifi-flow-registry-client-nar is returned to lib, and nifi is > restarted, these changes remain and registry is unreachable using this > registry client. > Update: The remainder of this ticket (that has had its text struck through) > has been made into its own ticket (NIFI-11570) as the underlying issue has > been found to be different. > -Also, if the nar removed was nifi-standard-services-api-nar, then besides > the above behavior, processors under version control via this registry client > may also have their dynamic properties encrypted. These properties remain > encrypted even after nifi-standard-services-api-nar is returned to lib and > nifi is restarted.- > -This is seen with a dynamic property added to GenerateFlowFile (when > GenericFlowFile is part of a PG under registry version control).- > -These are edge cases as admins should be very careful about removing nars > from lib, but it would be good if protections were added to protect > flow.xml/json from modifications in these situations.- -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11109) flow.json/xml modified when using registry client while missing nifi-flow-registry-client-nar
[ https://issues.apache.org/jira/browse/NIFI-11109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11109: - Status: Patch Available (was: Open) > flow.json/xml modified when using registry client while missing > nifi-flow-registry-client-nar > - > > Key: NIFI-11109 > URL: https://issues.apache.org/jira/browse/NIFI-11109 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > If nifi is set to use a registry client and nifi-flow-registry-client-nar is > removed from lib, the next nifi restart will result in the registry's class > name (in flow.xml.gz/flow.json.gz) to be modified from > org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient > to > NifiRegistryFlowRegistryClient. > The url property will also be encrypted. > When the nifi-flow-registry-client-nar is returned to lib, and nifi is > restarted, these changes remain and registry is unreachable using this > registry client. > Update: The remainder of this ticket (that has had its text struck through) > has been made into its own ticket (NIFI-11570) as the underlying issue has > been found to be different. > -Also, if the nar removed was nifi-standard-services-api-nar, then besides > the above behavior, processors under version control via this registry client > may also have their dynamic properties encrypted. These properties remain > encrypted even after nifi-standard-services-api-nar is returned to lib and > nifi is restarted.- > -This is seen with a dynamic property added to GenerateFlowFile (when > GenericFlowFile is part of a PG under registry version control).- > -These are edge cases as admins should be very careful about removing nars > from lib, but it would be good if protections were added to protect > flow.xml/json from modifications in these situations.- -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11109) flow.json/xml modified when using registry client while missing nifi-flow-registry-client-nar
[ https://issues.apache.org/jira/browse/NIFI-11109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11109: - Description: If nifi is set to use a registry client and nifi-flow-registry-client-nar is removed from lib, the next nifi restart will result in the registry's class name (in flow.xml.gz/flow.json.gz) to be modified from org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient to NifiRegistryFlowRegistryClient. The url property will also be encrypted. When the nifi-flow-registry-client-nar is returned to lib, and nifi is restarted, these changes remain and registry is unreachable using this registry client. Update: The remainder of this ticket (that has had its text struck through) has been made into its own ticket (NIFI-11570) as the underlying issue has been found to be different. -Also, if the nar removed was nifi-standard-services-api-nar, then besides the above behavior, processors under version control via this registry client may also have their dynamic properties encrypted. These properties remain encrypted even after nifi-standard-services-api-nar is returned to lib and nifi is restarted.- -This is seen with a dynamic property added to GenerateFlowFile (when GenericFlowFile is part of a PG under registry version control).- -These are edge cases as admins should be very careful about removing nars from lib, but it would be good if protections were added to protect flow.xml/json from modifications in these situations.- was: If nifi is set to use a registry client and nifi-flow-registry-client-nar is removed from lib, the next nifi restart will result in the registry's class name (in flow.xml.gz/flow.json.gz) to be modified from org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient to NifiRegistryFlowRegistryClient. The url property will also be encrypted. When the nifi-flow-registry-client-nar is returned to lib, and nifi is restarted, these changes remain and registry is unreachable using this registry client. -Also, if the nar removed was nifi-standard-services-api-nar, then besides the above behavior, processors under version control via this registry client may also have their dynamic properties encrypted. These properties remain encrypted even after nifi-standard-services-api-nar is returned to lib and nifi is restarted.- -This is seen with a dynamic property added to GenerateFlowFile (when GenericFlowFile is part of a PG under registry version control).- -These are edge cases as admins should be very careful about removing nars from lib, but it would be good if protections were added to protect flow.xml/json from modifications in these situations.- > flow.json/xml modified when using registry client while missing > nifi-flow-registry-client-nar > - > > Key: NIFI-11109 > URL: https://issues.apache.org/jira/browse/NIFI-11109 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > If nifi is set to use a registry client and nifi-flow-registry-client-nar is > removed from lib, the next nifi restart will result in the registry's class > name (in flow.xml.gz/flow.json.gz) to be modified from > org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient > to > NifiRegistryFlowRegistryClient. > The url property will also be encrypted. > When the nifi-flow-registry-client-nar is returned to lib, and nifi is > restarted, these changes remain and registry is unreachable using this > registry client. > Update: The remainder of this ticket (that has had its text struck through) > has been made into its own ticket (NIFI-11570) as the underlying issue has > been found to be different. > -Also, if the nar removed was nifi-standard-services-api-nar, then besides > the above behavior, processors under version control via this registry client > may also have their dynamic properties encrypted. These properties remain > encrypted even after nifi-standard-services-api-nar is returned to lib and > nifi is restarted.- > -This is seen with a dynamic property added to GenerateFlowFile (when > GenericFlowFile is part of a PG under registry version control).- > -These are edge cases as admins should be very careful about removing nars > from lib, but it would be good if protections were added to protect > flow.xml/json from modifications in these situations.- -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11109) flow.json/xml modified when using registry client while missing nifi-flow-registry-client-nar
[ https://issues.apache.org/jira/browse/NIFI-11109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11109: - Description: If nifi is set to use a registry client and nifi-flow-registry-client-nar is removed from lib, the next nifi restart will result in the registry's class name (in flow.xml.gz/flow.json.gz) to be modified from org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient to NifiRegistryFlowRegistryClient. The url property will also be encrypted. When the nifi-flow-registry-client-nar is returned to lib, and nifi is restarted, these changes remain and registry is unreachable using this registry client. -Also, if the nar removed was nifi-standard-services-api-nar, then besides the above behavior, processors under version control via this registry client may also have their dynamic properties encrypted. These properties remain encrypted even after nifi-standard-services-api-nar is returned to lib and nifi is restarted.- -This is seen with a dynamic property added to GenerateFlowFile (when GenericFlowFile is part of a PG under registry version control).- -These are edge cases as admins should be very careful about removing nars from lib, but it would be good if protections were added to protect flow.xml/json from modifications in these situations.- was: If nifi is set to use a registry client and nifi-flow-registry-client-nar is removed from lib, the next nifi restart will result in the registry's class name (in flow.xml.gz/flow.json.gz) to be modified from org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient to NifiRegistryFlowRegistryClient. The url property will also be encrypted. When the nifi-flow-registry-client-nar is returned to lib, and nifi is restarted, these changes remain and registry is unreachable using this registry client. Also, if the nar removed was nifi-standard-services-api-nar, then besides the above behavior, processors under version control via this registry client may also have their dynamic properties encrypted. These properties remain encrypted even after nifi-standard-services-api-nar is returned to lib and nifi is restarted. This is seen with a dynamic property added to GenerateFlowFile (when GenericFlowFile is part of a PG under registry version control). These are edge cases as admins should be very careful about removing nars from lib, but it would be good if protections were added to protect flow.xml/json from modifications in these situations. > flow.json/xml modified when using registry client while missing > nifi-flow-registry-client-nar > - > > Key: NIFI-11109 > URL: https://issues.apache.org/jira/browse/NIFI-11109 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > If nifi is set to use a registry client and nifi-flow-registry-client-nar is > removed from lib, the next nifi restart will result in the registry's class > name (in flow.xml.gz/flow.json.gz) to be modified from > org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient > to > NifiRegistryFlowRegistryClient. > The url property will also be encrypted. > When the nifi-flow-registry-client-nar is returned to lib, and nifi is > restarted, these changes remain and registry is unreachable using this > registry client. > -Also, if the nar removed was nifi-standard-services-api-nar, then besides > the above behavior, processors under version control via this registry client > may also have their dynamic properties encrypted. These properties remain > encrypted even after nifi-standard-services-api-nar is returned to lib and > nifi is restarted.- > -This is seen with a dynamic property added to GenerateFlowFile (when > GenericFlowFile is part of a PG under registry version control).- > -These are edge cases as admins should be very careful about removing nars > from lib, but it would be good if protections were added to protect > flow.xml/json from modifications in these situations.- -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11570) Processors have difficulty restoring dynamic properties when leaving ghost mode
Nissim Shiman created NIFI-11570: Summary: Processors have difficulty restoring dynamic properties when leaving ghost mode Key: NIFI-11570 URL: https://issues.apache.org/jira/browse/NIFI-11570 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.19.1 Reporter: Nissim Shiman Assignee: Nissim Shiman When a processor with non sensitive dynamic properties is started in ghost mode due to its underlying nar being missing, it will have a difficult time being restored even when its nar has been replaced. The dynamic properties will be identified as being sensitive and not able to be accessed. So for example, if a dynamic property added to GenerateFlowFile and nifi-standard-nar is removed from lib and nifi is restarted, GenerateFlowFile will come up in ghosted mode (as expected). When nifi-standard-nar is replaced and nifi is restarted, GenerateFlowFile will be restored, but its dynamic property will not be accessable. This was originally filed as a second part of NIFI-11109, but after looking more into the underlying causes/symptoms, realized that these are two different issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11463) Add option for IdentifyMimeType custom mime definitions to be in addition to the default tika definitions
[ https://issues.apache.org/jira/browse/NIFI-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-11463: Assignee: Nissim Shiman > Add option for IdentifyMimeType custom mime definitions to be in addition to > the default tika definitions > - > > Key: NIFI-11463 > URL: https://issues.apache.org/jira/browse/NIFI-11463 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > IdentifyMimeType has options for users to configure custom mime types > definitions. > These user defined definitions override the default tika mime type > definitions. > Add option to allow custom mime type definitions to be in addition to the > default tika mime type definitions. > Users will be able to decide if custom definitions should be used instead of > or in addition to default tika definitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11463) Add option for IdentifyMimeType custom mime definitions to be in addition to the default tika definitions
Nissim Shiman created NIFI-11463: Summary: Add option for IdentifyMimeType custom mime definitions to be in addition to the default tika definitions Key: NIFI-11463 URL: https://issues.apache.org/jira/browse/NIFI-11463 Project: Apache NiFi Issue Type: Improvement Reporter: Nissim Shiman IdentifyMimeType has options for users to configure custom mime types definitions. These user defined definitions override the default tika mime type definitions. Add option to allow custom mime type definitions to be in addition to the default tika mime type definitions. Users will be able to decide if custom definitions should be used instead of or in addition to default tika definitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11389) Controller Services's link to referencing Controller Services components not always working
[ https://issues.apache.org/jira/browse/NIFI-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11389: - Affects Version/s: 1.19.1 1.20.0 > Controller Services's link to referencing Controller Services components not > always working > --- > > Key: NIFI-11389 > URL: https://issues.apache.org/jira/browse/NIFI-11389 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0, 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > Controller Services' Setting tab has a _Referencing Components_ section on > the right side. > There are hyperlinks to the Processors and/or Controller Services which > reference this controller service. These links do not always work when > referencing controller services. > To reproduce: > Create StandardRestrictedSSLContextService on top level nifi graph (i.e. > right click top level grid -> Configure -> Controller Services) > Create Process Group > Inside Process Group create DistributedMapCacheServer > and point _SSL Context Service_ to StandardRestrictedSSLContextService just > created > Go to StandardRestrictedSSLContextService's Setting's tab and verify > DistributedMapCacheServer appears in the _Referencing Components_ section. > Clicking on the link to DistributedMapCacheServer will not take us to that > controller service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11389) Controller Services's link to referencing Controller Services components not always working
[ https://issues.apache.org/jira/browse/NIFI-11389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-11389: Assignee: Nissim Shiman > Controller Services's link to referencing Controller Services components not > always working > --- > > Key: NIFI-11389 > URL: https://issues.apache.org/jira/browse/NIFI-11389 > Project: Apache NiFi > Issue Type: Bug >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > Controller Services' Setting tab has a _Referencing Components_ section on > the right side. > There are hyperlinks to the Processors and/or Controller Services which > reference this controller service. These links do not always work when > referencing controller services. > To reproduce: > Create StandardRestrictedSSLContextService on top level nifi graph (i.e. > right click top level grid -> Configure -> Controller Services) > Create Process Group > Inside Process Group create DistributedMapCacheServer > and point _SSL Context Service_ to StandardRestrictedSSLContextService just > created > Go to StandardRestrictedSSLContextService's Setting's tab and verify > DistributedMapCacheServer appears in the _Referencing Components_ section. > Clicking on the link to DistributedMapCacheServer will not take us to that > controller service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11389) Controller Services's link to referencing Controller Services components not always working
Nissim Shiman created NIFI-11389: Summary: Controller Services's link to referencing Controller Services components not always working Key: NIFI-11389 URL: https://issues.apache.org/jira/browse/NIFI-11389 Project: Apache NiFi Issue Type: Bug Reporter: Nissim Shiman Controller Services' Setting tab has a _Referencing Components_ section on the right side. There are hyperlinks to the Processors and/or Controller Services which reference this controller service. These links do not always work when referencing controller services. To reproduce: Create StandardRestrictedSSLContextService on top level nifi graph (i.e. right click top level grid -> Configure -> Controller Services) Create Process Group Inside Process Group create DistributedMapCacheServer and point _SSL Context Service_ to StandardRestrictedSSLContextService just created Go to StandardRestrictedSSLContextService's Setting's tab and verify DistributedMapCacheServer appears in the _Referencing Components_ section. Clicking on the link to DistributedMapCacheServer will not take us to that controller service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11388) Backpressure queue settings loosely followed
[ https://issues.apache.org/jira/browse/NIFI-11388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709041#comment-17709041 ] Nissim Shiman commented on NIFI-11388: -- Thank you [~joewitt] and [~pvillard] for your responses. I noticed this when testing the current 1.21.0 RC2, but the queue did always stop after a moment ot two anyway so the basic idea of queues having backpressure is certainly working. Thank you for letting me know the current way it is working is by design. This works for me. Thank You! > Backpressure queue settings loosely followed > > > Key: NIFI-11388 > URL: https://issues.apache.org/jira/browse/NIFI-11388 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nissim Shiman >Priority: Major > > The backpressure settings on connections between processors are only loosely > followed. More flowfiles can end up on queues than configured via > backpressure setting. > For example, set up flow: > GenerateFlowFile -> UpdateAttribute -> (some processor) > where > GenerateFlowFile has _Custom Text_ set to be _hello_ > and Run Schedule is set to be _0 min_ > and > the connection following UpdateAttribute has _Back Pressure Object Threshold_ > set to be _100_ > Start GenerateFlowFile. > Wait a few moments until outgoing connection fills up. > Start UpdateAttribute > OutGoing conection will have more than 100 flowfiles on it > (I had over 1000 on mine when running these steps) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11388) Backpressure queue settings loosely followed
Nissim Shiman created NIFI-11388: Summary: Backpressure queue settings loosely followed Key: NIFI-11388 URL: https://issues.apache.org/jira/browse/NIFI-11388 Project: Apache NiFi Issue Type: Improvement Reporter: Nissim Shiman The backpressure settings on connections between processors are only loosely followed. More flowfiles can end up on queues than configured via backpressure setting. For example, set up flow: GenerateFlowFile -> UpdateAttribute -> (some processor) where GenerateFlowFile has _Custom Text_ set to be _hello_ and Run Schedule is set to be _0 min_ and the connection following UpdateAttribute has _Back Pressure Object Threshold_ set to be _100_ Start GenerateFlowFile. Wait a few moments until outgoing connection fills up. Start UpdateAttribute OutGoing conection will have more than 100 flowfiles on it (I had over 1000 on mine when running these steps) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-8710) StandardProvenanceReporter - receive() incorrectly overiding details parameter
[ https://issues.apache.org/jira/browse/NIFI-8710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707984#comment-17707984 ] Nissim Shiman commented on NIFI-8710: - Thank you [~mattyb149] for looping back to this to get into 2.0. I appreciate it. > StandardProvenanceReporter - receive() incorrectly overiding details parameter > -- > > Key: NIFI-8710 > URL: https://issues.apache.org/jira/browse/NIFI-8710 > Project: Apache NiFi > Issue Type: Bug >Reporter: Nissim Shiman >Assignee: Matt Burgess >Priority: Major > Fix For: 2.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The ProvenanceReporter interface has receive() with details as the third > parameter [1]: > void receive(FlowFile flowFile, String transitUri, String details, long > transmissionMillis); > StandardProvenanceReporter.java implements this with > sourceSystemFlowFileIdentifier as the third parameter [2]: > public void receive(final FlowFile flowFile, final String transitUri, final > String sourceSystemFlowFileIdentifier, final long transmissionMillis) > This implementation in StandardProvenanceReporter should be modified to > reflect the interface. > > [1] > [https://github.com/apache/nifi/blob/main/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L100] > [2] > [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/repository/StandardProvenanceReporter.java#L159] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10756) StandardSSLContextService will fail silently on nifi restart if keystore no longer exists
[ https://issues.apache.org/jira/browse/NIFI-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-10756: - Affects Version/s: 1.20.0 Status: Patch Available (was: Open) > StandardSSLContextService will fail silently on nifi restart if keystore no > longer exists > - > > Key: NIFI-10756 > URL: https://issues.apache.org/jira/browse/NIFI-10756 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0, 1.17.0 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > When StandardSSLContextService is successfully enabled, it will fail silently > on nifi restart if the keystore filename no longer exists. > There are no logs in nifi-app.log at WARN/ERROR that indicate any issues. > There is an alert in the nifi gui when looking at the controller service that > indicates that the needed file is missing. > Also, when debug is set via logback.xml, the issue can be seen as nifi will > continue to try to start it [1] > [1] > https://github.com/apache/nifi/blob/rel/nifi-1.17.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceNode.java#L586-L587 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-7123) onPropertyModified() is called on nifi start up even when properties have never been modified
[ https://issues.apache.org/jira/browse/NIFI-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17703273#comment-17703273 ] Nissim Shiman commented on NIFI-7123: - Thank you [~markap14] for the background information and the best practices pattern to deal with situation. At this point this is looking like maybe a tweak to the api document just to clarify that it is meant to always run on startup. Thank you for your comment! > onPropertyModified() is called on nifi start up even when properties have > never been modified > - > > Key: NIFI-7123 > URL: https://issues.apache.org/jira/browse/NIFI-7123 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Nissim Shiman >Priority: Major > > Processors and Controller Services inherit the onPropertyModified() method > from ConfigurableComponent. java [1] > This method is called when nifi starts for all processors and controller > services, even for properties that are set to their defaults (i.e. have never > been modified). > [1] > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/components/ConfigurableComponent.java#L68 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-7123) onPropertyModified() is called on nifi start up even when properties have never been modified
[ https://issues.apache.org/jira/browse/NIFI-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-7123: --- Assignee: (was: Nissim Shiman) > onPropertyModified() is called on nifi start up even when properties have > never been modified > - > > Key: NIFI-7123 > URL: https://issues.apache.org/jira/browse/NIFI-7123 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Nissim Shiman >Priority: Major > > Processors and Controller Services inherit the onPropertyModified() method > from ConfigurableComponent. java [1] > This method is called when nifi starts for all processors and controller > services, even for properties that are set to their defaults (i.e. have never > been modified). > [1] > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/components/ConfigurableComponent.java#L68 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-11166) IdentifyMimeType processor identifies flowfile-v3 as video/x-ms-wmv when containing wmv file
[ https://issues.apache.org/jira/browse/NIFI-11166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman resolved NIFI-11166. -- Resolution: Duplicate > IdentifyMimeType processor identifies flowfile-v3 as video/x-ms-wmv when > containing wmv file > > > Key: NIFI-11166 > URL: https://issues.apache.org/jira/browse/NIFI-11166 > Project: Apache NiFi > Issue Type: Bug >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > To recreate: > GetFile -> MergeContent -> IdentifyMimeType -> funnel > where GetFile is getting a .wmv video file. > MergeContent should have > Merge Format as "FlowFile Stream, v3" > Output flowfile will have mime.type of > video/x-ms-wmv > as opposed to > application/flowfile-v3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-4718) IdentifyMimeType: increase priority for FFv3
[
https://issues.apache.org/jira/browse/NIFI-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-4718:
Status: Patch Available (was: Open)
> IdentifyMimeType: increase priority for FFv3
>
>
> Key: NIFI-4718
> URL: https://issues.apache.org/jira/browse/NIFI-4718
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Brandon Rhys DeVries
>Assignee: Nissim Shiman
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> IdentifyMimeType uses tika configured with a custom-mimetypes.xml\[1] to
> specify (among others) the flowfile-v* mime types. However, these do not
> include priorities. Therefore, a NiFi FlowFile V3 package with a payload
> containing, for example, html including the string:
> {code}
> {code}
> will be identified as "application/xhtml+xml" \[2] which, while matching the
> pattern, is not as correct as identifying it as application/flowfile-v3. To
> fix this, I believe we need to specify a higher priority for the FlowFile V3
> "magic"...
> \[1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/resources/org/apache/tika/mime/custom-mimetypes.xml#L26-L31
> \[2]
> https://gitbox.apache.org/repos/asf?p=tika.git;a=blob;f=tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml;hb=refs/heads/master
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-4718) IdentifyMimeType: increase priority for FFv3
[
https://issues.apache.org/jira/browse/NIFI-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman reassigned NIFI-4718:
---
Assignee: Nissim Shiman
> IdentifyMimeType: increase priority for FFv3
>
>
> Key: NIFI-4718
> URL: https://issues.apache.org/jira/browse/NIFI-4718
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Brandon Rhys DeVries
>Assignee: Nissim Shiman
>Priority: Minor
>
> IdentifyMimeType uses tika configured with a custom-mimetypes.xml\[1] to
> specify (among others) the flowfile-v* mime types. However, these do not
> include priorities. Therefore, a NiFi FlowFile V3 package with a payload
> containing, for example, html including the string:
> {code}
> {code}
> will be identified as "application/xhtml+xml" \[2] which, while matching the
> pattern, is not as correct as identifying it as application/flowfile-v3. To
> fix this, I believe we need to specify a higher priority for the FlowFile V3
> "magic"...
> \[1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/resources/org/apache/tika/mime/custom-mimetypes.xml#L26-L31
> \[2]
> https://gitbox.apache.org/repos/asf?p=tika.git;a=blob;f=tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml;hb=refs/heads/master
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-11109) flow.json/xml modified when using registry client while missing nifi-flow-registry-client-nar
[ https://issues.apache.org/jira/browse/NIFI-11109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-11109: Assignee: Nissim Shiman > flow.json/xml modified when using registry client while missing > nifi-flow-registry-client-nar > - > > Key: NIFI-11109 > URL: https://issues.apache.org/jira/browse/NIFI-11109 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.19.1 >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > If nifi is set to use a registry client and nifi-flow-registry-client-nar is > removed from lib, the next nifi restart will result in the registry's class > name (in flow.xml.gz/flow.json.gz) to be modified from > org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient > to > NifiRegistryFlowRegistryClient. > The url property will also be encrypted. > When the nifi-flow-registry-client-nar is returned to lib, and nifi is > restarted, these changes remain and registry is unreachable using this > registry client. > Also, if the nar removed was nifi-standard-services-api-nar, then besides the > above behavior, processors under version control via this registry client may > also have their dynamic properties encrypted. These properties remain > encrypted even after nifi-standard-services-api-nar is returned to lib and > nifi is restarted. > This is seen with a dynamic property added to GenerateFlowFile (when > GenericFlowFile is part of a PG under registry version control). > These are edge cases as admins should be very careful about removing nars > from lib, but it would be good if protections were added to protect > flow.xml/json from modifications in these situations. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11166) IdentifyMimeType processor identifies flowfile-v3 as video/x-ms-wmv when containing wmv file
[ https://issues.apache.org/jira/browse/NIFI-11166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-11166: Assignee: Nissim Shiman > IdentifyMimeType processor identifies flowfile-v3 as video/x-ms-wmv when > containing wmv file > > > Key: NIFI-11166 > URL: https://issues.apache.org/jira/browse/NIFI-11166 > Project: Apache NiFi > Issue Type: Bug >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > To recreate: > GetFile -> MergeContent -> IdentifyMimeType -> funnel > where GetFile is getting a .wmv video file. > MergeContent should have > Merge Format as "FlowFile Stream, v3" > Output flowfile will have mime.type of > video/x-ms-wmv > as opposed to > application/flowfile-v3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11166) IdentifyMimeType processor identifies flowfile-v3 as video/x-ms-wmv when containing wmv file
Nissim Shiman created NIFI-11166: Summary: IdentifyMimeType processor identifies flowfile-v3 as video/x-ms-wmv when containing wmv file Key: NIFI-11166 URL: https://issues.apache.org/jira/browse/NIFI-11166 Project: Apache NiFi Issue Type: Bug Reporter: Nissim Shiman To recreate: GetFile -> MergeContent -> IdentifyMimeType -> funnel where GetFile is getting a .wmv video file. MergeContent should have Merge Format as "FlowFile Stream, v3" Output flowfile will have mime.type of video/x-ms-wmv as opposed to application/flowfile-v3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11153) For very large files, content viewer fails with OutOfMemoryError, screen looses look and feel
Nissim Shiman created NIFI-11153: Summary: For very large files, content viewer fails with OutOfMemoryError, screen looses look and feel Key: NIFI-11153 URL: https://issues.apache.org/jira/browse/NIFI-11153 Project: Apache NiFi Issue Type: Bug Reporter: Nissim Shiman For very large files, content viewer fails with HTTP ERROR 500 java.lang.OutOfMemoryError: Java heap space The nifi gui loses its "look and feel" and the screen goes white with the additional details: URI: /nifi-content-viewer/ STATUS: 500 MESSAGE: java.lang.OutOfMemoryError: Java heap space SERVLET: ContentViewerController CAUSED BY: java.lang.OutOfMemoryError: Java heap space Caused by: java.lang.OutOfMemoryError: Java heap space To recreate: GenerateFlowFile -> funnel where GenerateFlowFile's filesize is 100MB then view file on output queue (Issue was also seen with json file of 100MB) Possible solution maybe to check for file over a certain size before attempting to process for viewing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10950) Distribute Load processor - remove Load Distribution Service as Distribution Strategy
[
https://issues.apache.org/jira/browse/NIFI-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-10950:
-
Status: Patch Available (was: In Progress)
> Distribute Load processor - remove Load Distribution Service as Distribution
> Strategy
> -
>
> Key: NIFI-10950
> URL: https://issues.apache.org/jira/browse/NIFI-10950
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The DistributeLoad processor's Distribution Strategy property is currently
> able to be set to "load distribution service"
> But when attempting to create the load distribution service, we get the
> message:
> {code:java}
> No controller service types found that are applicable for this property
> {code}
> The underlying load distribution service api [1] is not implemented, so it
> can be removed as a distribution strategy, in order to avoid the above
> situation.
> [1]
> [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-standard-services/nifi-load-distribution-service-api/src/main/java/org/apache/nifi/loading/LoadDistributionService.java]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-10878) nifi unable to start off of flow.xml.gz when pointing to registry client
[ https://issues.apache.org/jira/browse/NIFI-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman resolved NIFI-10878. -- Resolution: Fixed > nifi unable to start off of flow.xml.gz when pointing to registry client > > > Key: NIFI-10878 > URL: https://issues.apache.org/jira/browse/NIFI-10878 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.18.0 >Reporter: Nissim Shiman >Priority: Major > > If nifi is using a registry client and is started from a flow.xml.gz file (as > opposed to flow.json.gz) then it will be unable to start. > This is more of a backward compatibility related concern since flow.json.gz > is now the recommended flow configuration file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10878) nifi unable to start off of flow.xml.gz when pointing to registry client
[ https://issues.apache.org/jira/browse/NIFI-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682112#comment-17682112 ] Nissim Shiman commented on NIFI-10878: -- This was resolved with NIFI-10937. > nifi unable to start off of flow.xml.gz when pointing to registry client > > > Key: NIFI-10878 > URL: https://issues.apache.org/jira/browse/NIFI-10878 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.18.0 >Reporter: Nissim Shiman >Priority: Major > > If nifi is using a registry client and is started from a flow.xml.gz file (as > opposed to flow.json.gz) then it will be unable to start. > This is more of a backward compatibility related concern since flow.json.gz > is now the recommended flow configuration file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11109) flow.json/xml modified when using registry client while missing nifi-flow-registry-client-nar
[ https://issues.apache.org/jira/browse/NIFI-11109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11109: - Affects Version/s: 1.19.1 > flow.json/xml modified when using registry client while missing > nifi-flow-registry-client-nar > - > > Key: NIFI-11109 > URL: https://issues.apache.org/jira/browse/NIFI-11109 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.19.1 >Reporter: Nissim Shiman >Priority: Major > > If nifi is set to use a registry client and nifi-flow-registry-client-nar is > removed from lib, the next nifi restart will result in the registry's class > name (in flow.xml.gz/flow.json.gz) to be modified from > org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient > to > NifiRegistryFlowRegistryClient. > The url property will also be encrypted. > When the nifi-flow-registry-client-nar is returned to lib, and nifi is > restarted, these changes remain and registry is unreachable using this > registry client. > Also, if the nar removed was nifi-standard-services-api-nar, then besides the > above behavior, processors under version control via this registry client may > also have their dynamic properties encrypted. These properties remain > encrypted even after nifi-standard-services-api-nar is returned to lib and > nifi is restarted. > This is seen with a dynamic property added to GenerateFlowFile (when > GenericFlowFile is part of a PG under registry version control). > These are edge cases as admins should be very careful about removing nars > from lib, but it would be good if protections were added to protect > flow.xml/json from modifications in these situations. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11109) flow.json/xml modified when using registry client while missing nifi-flow-registry-client-nar
Nissim Shiman created NIFI-11109: Summary: flow.json/xml modified when using registry client while missing nifi-flow-registry-client-nar Key: NIFI-11109 URL: https://issues.apache.org/jira/browse/NIFI-11109 Project: Apache NiFi Issue Type: Bug Reporter: Nissim Shiman If nifi is set to use a registry client and nifi-flow-registry-client-nar is removed from lib, the next nifi restart will result in the registry's class name (in flow.xml.gz/flow.json.gz) to be modified from org.apache.nifi.registry.flow.NifiRegistryFlowRegistryClient to NifiRegistryFlowRegistryClient. The url property will also be encrypted. When the nifi-flow-registry-client-nar is returned to lib, and nifi is restarted, these changes remain and registry is unreachable using this registry client. Also, if the nar removed was nifi-standard-services-api-nar, then besides the above behavior, processors under version control via this registry client may also have their dynamic properties encrypted. These properties remain encrypted even after nifi-standard-services-api-nar is returned to lib and nifi is restarted. This is seen with a dynamic property added to GenerateFlowFile (when GenericFlowFile is part of a PG under registry version control). These are edge cases as admins should be very careful about removing nars from lib, but it would be good if protections were added to protect flow.xml/json from modifications in these situations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11041) Refactor nifi registry to use JUnit5 - Part1
[ https://issues.apache.org/jira/browse/NIFI-11041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11041: - Status: Patch Available (was: Open) > Refactor nifi registry to use JUnit5 - Part1 > > > Key: NIFI-11041 > URL: https://issues.apache.org/jira/browse/NIFI-11041 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This covers the smaller folders (i.e. everything except > nifi-registry-framework, nifi-registry-web-api, nifi-registry-web-ui folders). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11041) Refactor nifi registry to use JUnit5 - Part1
[ https://issues.apache.org/jira/browse/NIFI-11041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11041: - Description: This covers the smaller folders (i.e. everything except nifi-registry-framework, nifi-registry-web-api, nifi-registry-web-ui folders). (was: This covers the nifi-registry-bundle-utils, nifi-registry-client, and nifi-registry-data-model directories.) > Refactor nifi registry to use JUnit5 - Part1 > > > Key: NIFI-11041 > URL: https://issues.apache.org/jira/browse/NIFI-11041 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > This covers the smaller folders (i.e. everything except > nifi-registry-framework, nifi-registry-web-api, nifi-registry-web-ui folders). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11041) Refactor nifi registry to use JUnit5 - Part1
[ https://issues.apache.org/jira/browse/NIFI-11041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-11041: Assignee: Nissim Shiman > Refactor nifi registry to use JUnit5 - Part1 > > > Key: NIFI-11041 > URL: https://issues.apache.org/jira/browse/NIFI-11041 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > This covers the nifi-registry-bundle-utils, nifi-registry-client, and > nifi-registry-data-model directories. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11041) Refactor nifi registry to use JUnit5 - Part1
Nissim Shiman created NIFI-11041: Summary: Refactor nifi registry to use JUnit5 - Part1 Key: NIFI-11041 URL: https://issues.apache.org/jira/browse/NIFI-11041 Project: Apache NiFi Issue Type: Sub-task Reporter: Nissim Shiman This covers the nifi-registry-bundle-utils, nifi-registry-client, and nifi-registry-data-model directories. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10772) Unattributable error on nifi shutdown when controller service was unable to be started
[
https://issues.apache.org/jira/browse/NIFI-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-10772:
-
Status: Patch Available (was: Open)
> Unattributable error on nifi shutdown when controller service was unable to
> be started
> --
>
> Key: NIFI-10772
> URL: https://issues.apache.org/jira/browse/NIFI-10772
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This error occurs when nifi is unable to start a controller service that is
> supposed to be in an enabled state. On shutdown, nifi will give an error
> (stacktrace below)
> To reproduce, for example using, StandardRestrictedSSLContextService:
> Enable StandardRestrictedSSLContextService
> Shutdown nifi
> remove keystore StandardRestrictedSSLContextService relied on (or move it to
> different location on filesystem)
> start nifi
> stop nifi
> When nifi is shutdown the following uncaught/non-attributable error is in
> nifi-app.log:
> {code:java}
> 2023-01-06 15:46:41,085 ERROR [Timer-Driven Process Thread-5]
> org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2867c735
> rejected from org.apache.nifi.e
> ngine.FlowEngine@a814d7d[Shutting down, pool size = 10, active threads = 3,
> queued tasks = 0, completed tasks = 257823]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
> at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
> at
> org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> It is unclear from the current log output as to what the underlying cause of
> it was (i.e. which controller service StandardControllerServiceNode is having
> trouble with)
> A similar non-attributable error is also seen on nifi shutdown for a
> processor that relies on this controller service.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-11030) Refactor nifi-stateless-processor-bundle to use JUnit5
Nissim Shiman created NIFI-11030: Summary: Refactor nifi-stateless-processor-bundle to use JUnit5 Key: NIFI-11030 URL: https://issues.apache.org/jira/browse/NIFI-11030 Project: Apache NiFi Issue Type: Sub-task Reporter: Nissim Shiman Assignee: Nissim Shiman -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-10772) Unattributable error on nifi shutdown when controller service was unable to be started
[
https://issues.apache.org/jira/browse/NIFI-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655509#comment-17655509
]
Nissim Shiman commented on NIFI-10772:
--
Here is the stack trace (also uncaught/unattributable) on nifi shutdown when a
processor relies on a controller service in the state mention above
{code:java}
2023-01-06 15:46:40,492 ERROR [Monitor Processor Lifecycle Thread-2]
org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@398f32ee
rejected from org.apache.nifi.engine.FlowEngine@a814d7d[Shutting down, pool
size = 10, active threads = 0, queued tasks = 3, completed tasks = 257823]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:549)
at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:102)
at
java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:648)
at
org.apache.nifi.controller.scheduling.StandardProcessScheduler$4.scheduleTask(StandardProcessScheduler.java:350)
at
org.apache.nifi.controller.StandardProcessorNode.initiateStart(StandardProcessorNode.java:1803)
at
org.apache.nifi.controller.StandardProcessorNode.lambda$null$6(StandardProcessorNode.java:1715)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
To see this, do the following steps (similar to those in ticket description
above):
Enable StandardRestrictedSSLContextService
Create ListenHTTP processor
set processor's SSL Context Service to use StandardRestrictedSSLContextService
just enabled
Start ListenHTTP
Shutdown nifi
remove keystore StandardRestrictedSSLContextService relied on (or move it to
different location on filesystem)
start nifi
stop nifi
Two errors will be seen, one for the controller service and one for the
processor.
> Unattributable error on nifi shutdown when controller service was unable to
> be started
> --
>
> Key: NIFI-10772
> URL: https://issues.apache.org/jira/browse/NIFI-10772
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
>
> This error occurs when nifi is unable to start a controller service that is
> supposed to be in an enabled state. On shutdown, nifi will give an error
> (stacktrace below)
> To reproduce, for example using, StandardRestrictedSSLContextService:
> Enable StandardRestrictedSSLContextService
> Shutdown nifi
> remove keystore StandardRestrictedSSLContextService relied on (or move it to
> different location on filesystem)
> start nifi
> stop nifi
> When nifi is shutdown the following uncaught/non-attributable error is in
> nifi-app.log:
> {code:java}
> 2023-01-06 15:46:41,085 ERROR [Timer-Driven Process Thread-5]
> org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2867c735
> rejected from org.apache.nifi.e
> ngine.FlowEngine@a814d7d[Shutting down, pool size = 10, active threads = 3,
> queued tasks = 0, completed tasks = 257823]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
> at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
>
[jira] [Updated] (NIFI-10772) Unattributable error on nifi shutdown when controller service was unable to be started
[
https://issues.apache.org/jira/browse/NIFI-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-10772:
-
Description:
This error occurs when nifi is unable to start a controller service that is
supposed to be in an enabled state. On shutdown, nifi will give an error
(stacktrace below)
To reproduce, for example using, StandardRestrictedSSLContextService:
Enable StandardRestrictedSSLContextService
Shutdown nifi
remove keystore StandardRestrictedSSLContextService relied on (or move it to
different location on filesystem)
start nifi
stop nifi
When nifi is shutdown the following uncaught/non-attributable error is in
nifi-app.log:
{code:java}
2023-01-06 15:46:41,085 ERROR [Timer-Driven Process Thread-5]
org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2867c735
rejected from org.apache.nifi.e
ngine.FlowEngine@a814d7d[Shutting down, pool size = 10, active threads = 3,
queued tasks = 0, completed tasks = 257823]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
at
org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
It is unclear from the current log output as to what the underlying cause of it
was (i.e. which controller service StandardControllerServiceNode is having
trouble with)
A similar non-attributable error is also seen on nifi shutdown for a processor
that relies on this controller service.
was:
This error occurs when nifi is unable to start a controller service that is
supposed to be in an enabled state. On shutdown, nifi will give an error
(stacktrace below)
To reproduce, for example using, StandardRestrictedSSLContextService:
Enable StandardRestrictedSSLContextService
Shutdown nifi
remove keystore StandardRestrictedSSLContextService relied on (or move it to
different location on filesystem)
start nifi
stop nifi
When nifi is shutdown the following non-attributable error is in nifi-app.log:
{code:java}
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1c5f3492
rejected from org.apache.nifi.engine.FlowEngine@1298c969[Shutting down, pool
size = 10, active threads = 1, queued tasks = 0, completed tasks = 54]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
at
org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
It is unclear from the current log output as to
[jira] [Updated] (NIFI-10772) Unattributable error on nifi shutdown when controller service was unable to be started
[
https://issues.apache.org/jira/browse/NIFI-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-10772:
-
Description:
This error occurs when nifi is unable to start a controller service that is
supposed to be in an enabled state. On shutdown, nifi will give an error
(stacktrace below)
To reproduce, for example using, StandardRestrictedSSLContextService:
Enable StandardRestrictedSSLContextService
Shutdown nifi
remove keystore StandardRestrictedSSLContextService relied on (or move it to
different location on filesystem)
start nifi
stop nifi
When nifi is shutdown the following non-attributable error is in nifi-app.log:
{code:java}
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1c5f3492
rejected from org.apache.nifi.engine.FlowEngine@1298c969[Shutting down, pool
size = 10, active threads = 1, queued tasks = 0, completed tasks = 54]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
at
org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
It is unclear from the current log output as to what the underlying cause of it
was (i.e. which controller service StandardControllerServiceNode is having
trouble with)
A similar non-attributable error is also seen on nifi shutdown for a processor
that relies on this controller service.
was:
This error occurs when nifi is unable to start a controller service that is
supposed to be in an enabled state. On shutdown, nifi will give an error
(stacktrace below)
To reproduce, for example using, StandardSSLContextService:
Enable StandardSSLContextService
Shutdown nifi
remove keystore StandardSSLContextService relied on (or move it to different
location on filesystem)
start nifi
stop nifi
When nifi is shutdown the following non-attributable error is in nifi-app.log:
{code:java}
2023-01-06 15:46:41,085 ERROR [Timer-Driven Process Thread-5]
org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2867c735
rejected from org.apache.nifi.e
ngine.FlowEngine@a814d7d[Shutting down, pool size = 10, active threads = 3,
queued tasks = 0, completed tasks = 257823]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
at
org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
It is unclear from the current log output as to what the underlying cause of it
was
[jira] [Updated] (NIFI-10772) Unattributable error on nifi shutdown when controller service was unable to be started
[
https://issues.apache.org/jira/browse/NIFI-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-10772:
-
Description:
This error occurs when nifi is unable to start a controller service that is
supposed to be in an enabled state. On shutdown, nifi will give an error
(stacktrace below)
To reproduce, for example using, StandardSSLContextService:
Enable StandardSSLContextService
Shutdown nifi
remove keystore StandardSSLContextService relied on (or move it to different
location on filesystem)
start nifi
stop nifi
When nifi is shutdown the following non-attributable error is in nifi-app.log:
{code:java}
2023-01-06 15:46:41,085 ERROR [Timer-Driven Process Thread-5]
org.apache.nifi.engine.FlowEngine Uncaught Exception in Runnable task
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2867c735
rejected from org.apache.nifi.e
ngine.FlowEngine@a814d7d[Shutting down, pool size = 10, active threads = 3,
queued tasks = 0, completed tasks = 257823]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
at
org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
It is unclear from the current log output as to what the underlying cause of it
was (i.e. which controller service StandardControllerServiceNode is having
trouble with)
A similar non-attributable error is also seen on nifi shutdown for a processor
that relies on this controller service.
was:
This error occurs when nifi is unable to start a controller service that is
supposed to be in an enabled state. On shutdown, nifi will give an error
(stacktrace below)
To reproduce, for example using, StandardSSLContextService:
Enable StandardSSLContextService
Shutdown nifi
remove keystore StandardSSLContextService relied on (or move it to different
location on filesystem)
start nifi
stop nifi
When nifi is shutdown the following non-attributable error is in nifi-app.log:
{code:java}
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1c5f3492
rejected from org.apache.nifi.engine.FlowEngine@1298c969[Shutting down, pool
size = 10, active threads = 1, queued tasks = 0, completed tasks = 54]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
at
org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
It is unclear from the current log output as to what the underlying cause of it
was (i.e. which controller
[jira] [Updated] (NIFI-10772) Unattributable error on nifi shutdown when controller service was unable to be started
[
https://issues.apache.org/jira/browse/NIFI-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-10772:
-
Affects Version/s: 1.20.0
> Unattributable error on nifi shutdown when controller service was unable to
> be started
> --
>
> Key: NIFI-10772
> URL: https://issues.apache.org/jira/browse/NIFI-10772
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
>
> This error occurs when nifi is unable to start a controller service that is
> supposed to be in an enabled state. On shutdown, nifi will give an error
> (stacktrace below)
> To reproduce, for example using, StandardSSLContextService:
> Enable StandardSSLContextService
> Shutdown nifi
> remove keystore StandardSSLContextService relied on (or move it to different
> location on filesystem)
> start nifi
> stop nifi
> When nifi is shutdown the following non-attributable error is in nifi-app.log:
> {code:java}
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1c5f3492
> rejected from org.apache.nifi.engine.FlowEngine@1298c969[Shutting down, pool
> size = 10, active threads = 1, queued tasks = 0, completed tasks = 54]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
> at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
> at
> org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> It is unclear from the current log output as to what the underlying cause of
> it was (i.e. which controller service StandardControllerServiceNode is having
> trouble with)
> A similar non-attributable error is also seen on nifi shutdown for a
> processor that relies on this controller service.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10772) Unattributable error on nifi shutdown when controller service was unable to be started
[
https://issues.apache.org/jira/browse/NIFI-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nissim Shiman updated NIFI-10772:
-
Description:
This error occurs when nifi is unable to start a controller service that is
supposed to be in an enabled state. On shutdown, nifi will give an error
(stacktrace below)
To reproduce, for example using, StandardSSLContextService:
Enable StandardSSLContextService
Shutdown nifi
remove keystore StandardSSLContextService relied on (or move it to different
location on filesystem)
start nifi
stop nifi
When nifi is shutdown the following non-attributable error is in nifi-app.log:
{code:java}
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1c5f3492
rejected from org.apache.nifi.engine.FlowEngine@1298c969[Shutting down, pool
size = 10, active threads = 1, queued tasks = 0, completed tasks = 54]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
at
org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
It is unclear from the current log output as to what the underlying cause of it
was (i.e. which controller service StandardControllerServiceNode is having
trouble with)
A similar non-attributable error is also seen on nifi shutdown for a processor
that relies on this controller service.
was:
This error occurs when nifi is unable to start an controller service that is
supposed to be in an enabled state.
For example, if the StandardSSLContextService is running, but the underlying
keystore file it relies on is removed from the filesystem, nifi will be unable
to successfully start it on a nifi restart.
When nifi is shutdown the following error is in nifi-app.log:
{code}
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1c5f3492
rejected from org.apache.nifi.engine.FlowEngine@1298c969[Shutting down, pool
size = 10, active threads = 1, queued tasks = 0, completed tasks = 54]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at org.apache.nifi.engine.FlowEngine.schedule(FlowEngine.java:87)
at
org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:591)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
It is unclear from the current log output as to what the underlying cause of it
was (i.e. which controller service StandardControllerServiceNode is having
trouble with)
> Unattributable error on nifi shutdown when controller service was unable to
> be started
> --
>
>
[jira] [Updated] (NIFI-11019) Refactor nifi-scripting bundle to Junit 5 - Part 2
[ https://issues.apache.org/jira/browse/NIFI-11019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11019: - Status: Patch Available (was: Open) > Refactor nifi-scripting bundle to Junit 5 - Part 2 > -- > > Key: NIFI-11019 > URL: https://issues.apache.org/jira/browse/NIFI-11019 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This is a continuation of NIFI-9148 for a small number of files that look > like they were not yet in main when this was done. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11019) Refactor nifi-scripting bundle to Junit 5 - Part 2
[ https://issues.apache.org/jira/browse/NIFI-11019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman updated NIFI-11019: - Description: This is a continuation of NIFI-9148 for a small number of files that look like they were not yet in main when this was done. > Refactor nifi-scripting bundle to Junit 5 - Part 2 > -- > > Key: NIFI-11019 > URL: https://issues.apache.org/jira/browse/NIFI-11019 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Nissim Shiman >Priority: Major > > This is a continuation of NIFI-9148 for a small number of files that look > like they were not yet in main when this was done. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11019) Refactor nifi-scripting bundle to Junit 5 - Part 2
[ https://issues.apache.org/jira/browse/NIFI-11019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nissim Shiman reassigned NIFI-11019: Assignee: Nissim Shiman > Refactor nifi-scripting bundle to Junit 5 - Part 2 > -- > > Key: NIFI-11019 > URL: https://issues.apache.org/jira/browse/NIFI-11019 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > > This is a continuation of NIFI-9148 for a small number of files that look > like they were not yet in main when this was done. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11019) Refactor nifi-scripting bundle to Junit 5 - Part 2
Nissim Shiman created NIFI-11019: Summary: Refactor nifi-scripting bundle to Junit 5 - Part 2 Key: NIFI-11019 URL: https://issues.apache.org/jira/browse/NIFI-11019 Project: Apache NiFi Issue Type: Sub-task Reporter: Nissim Shiman -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-9164) Refactor nifi-windows-event-log-bundle to use JUnit 5
[ https://issues.apache.org/jira/browse/NIFI-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17652600#comment-17652600 ] Nissim Shiman edited comment on NIFI-9164 at 12/28/22 9:09 PM: --- This is trickier than the usual steps for junit4 to junit5 conversions as there are custom classloader workarounds that need to be done here [1][2] that fit in well with Junit 4 RunsWith, but do not have corresponding hooks for it in junit5. This is a known issue [3], with a fix currently targeted for the next release of junit5 (5.10.0-M1) [4] This being said, this issue has been known and discussed extensively for the past 6 years [5] , so it is unclear it will make it into junit 5, until it actually happens, although the recent refocus on it [3] is encouraging. [1] [https://github.com/apache/nifi/blob/rel/nifi-1.19.1/nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLogTest.java#L64] [2] [https://github.com/apache/nifi/blob/rel/nifi-1.19.1/nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/JNAJUnitRunner.java#L27] [3] [https://github.com/junit-team/junit5/issues/3028] [4] [https://github.com/junit-team/junit5/milestone/65] [5] [https://github.com/junit-team/junit5/issues/201] was (Author: nissim shiman): This is trickier than the usual steps for junit4 to junit5 conversions as there are custom classloader workarounds that need to be done here [1][2] that fit in well with Junit 4 RunsWith, but do not have corresponding hooks for it in junit5. This is a known issue [3], with a fix currently targeted for the next release of junit5 (5.10.0-M1) [4] This being said, this issue has been known and discussed extensively for the past 6 years [5] , so it is unclear it will make it into junit 5, until it actually happens, although the recent refocus on it [4] is encouraging. [1] [https://github.com/apache/nifi/blob/rel/nifi-1.19.1/nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLogTest.java#L64] [2] [https://github.com/apache/nifi/blob/rel/nifi-1.19.1/nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/JNAJUnitRunner.java#L27] [3] [https://github.com/junit-team/junit5/issues/3028] [4] [https://github.com/junit-team/junit5/milestone/65] [5] [https://github.com/junit-team/junit5/issues/201] > Refactor nifi-windows-event-log-bundle to use JUnit 5 > - > > Key: NIFI-9164 > URL: https://issues.apache.org/jira/browse/NIFI-9164 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Mike Thomsen >Assignee: Nissim Shiman >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-9164) Refactor nifi-windows-event-log-bundle to use JUnit 5
[ https://issues.apache.org/jira/browse/NIFI-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17652600#comment-17652600 ] Nissim Shiman commented on NIFI-9164: - This is trickier than the usual steps for junit4 to junit5 conversions as there are custom classloader workarounds that need to be done here [1][2] that fit in well with Junit 4 RunsWith, but do not have corresponding hooks for it in junit5. This is a known issue [3], with a fix currently targeted for the next release of junit5 (5.10.0-M1) [4] This being said, this issue has been known and discussed extensively for the past 6 years [5] , so it is unclear it will make it into junit 5, until it actually happens, although the recent refocus on it [4] is encouraging. [1] [https://github.com/apache/nifi/blob/rel/nifi-1.19.1/nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLogTest.java#L64] [2] [https://github.com/apache/nifi/blob/rel/nifi-1.19.1/nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/JNAJUnitRunner.java#L27] [3] [https://github.com/junit-team/junit5/issues/3028] [4] [https://github.com/junit-team/junit5/milestone/65] [5] [https://github.com/junit-team/junit5/issues/201] > Refactor nifi-windows-event-log-bundle to use JUnit 5 > - > > Key: NIFI-9164 > URL: https://issues.apache.org/jira/browse/NIFI-9164 > Project: Apache NiFi > Issue Type: Sub-task >Reporter: Mike Thomsen >Assignee: Nissim Shiman >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
