[jira] [Commented] (NIFI-5557) PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
[
https://issues.apache.org/jira/browse/NIFI-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623045#comment-16623045
]
ASF GitHub Bot commented on NIFI-5557:
--
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2971#discussion_r219380028
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
---
@@ -266,6 +271,16 @@ public Object run() {
throw new
IOException(configuredRootDirPath.toString() + " could not be created");
}
changeOwner(context, hdfs, configuredRootDirPath,
flowFile);
+} catch (IOException e) {
+ boolean tgtExpired = hasCause(e, GSSException.class,
gsse -> GSSException.NO_CRED == gsse.getMajor());
+ if (tgtExpired) {
+getLogger().error(String.format("An error occured
while connecting to HDFS. Rolling back session, and penalizing flow file %s",
--- End diff --
The exception be logged here, in addition to the flowfile UUID. It might
be useful to have the stack trace and exception class available in the log, and
we shouldn't suppress/omit the actual GSSException from the logging.
It might also be a good idea to log this at the "warn" level, so that the
user can choose to not have these show as bulletins on the processor in the UI.
Since the flowfile is being rolled back, and hadoop-client will implicitly
acquire a new ticket, I don't think this should show as an error. @mcgilman,
@bbende, do either of you have a preference here?
> PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
> -
>
> Key: NIFI-5557
> URL: https://issues.apache.org/jira/browse/NIFI-5557
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Endre Kovacs
>Assignee: Endre Kovacs
>Priority: Major
>
> when using *PutHDFS* processor in a kerberized environment, with a flow
> "traffic" which approximately matches or less frequent then the lifetime of
> the ticket of the principal, we see this in the log:
> {code:java}
> INFO [Timer-Driven Process Thread-4] o.a.h.io.retry.RetryInvocationHandler
> Exception while invoking getFileInfo of class
> ClientNamenodeProtocolTranslatorPB over host2/ip2:8020 after 13 fail over
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for princi...@example.com to host2.example.com/ip2:8020;
> Host Details : local host is: "host1.example.com/ip1"; destination host is:
> "host2.example.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> at sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy135.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:254)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
> at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:222)
> {code}
> and the flowfile is routed to failure relationship.
> *To reproduce:*
> Create a principal in your KDC with two minutes ticket lifetime,
> and set up a similar flow:
> {code:java}
> GetFile =>
[jira] [Commented] (NIFI-5557) PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
[
https://issues.apache.org/jira/browse/NIFI-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623046#comment-16623046
]
ASF GitHub Bot commented on NIFI-5557:
--
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2971#discussion_r219377632
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
---
@@ -266,6 +271,16 @@ public Object run() {
throw new
IOException(configuredRootDirPath.toString() + " could not be created");
}
changeOwner(context, hdfs, configuredRootDirPath,
flowFile);
+} catch (IOException e) {
--- End diff --
Thanks for changing this to use GSSException.getMajor(). I haven't tested
a ticket expiration occurring during the execution of a call to ugi.doAs (as
opposed to the ticket having expired before ugi.doAs is invoked), but I think
it would be a good idea to move this catch block to the top level try/catch
block of the PrivelegedExceptionAction passed to ugi.doAs().
> PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
> -
>
> Key: NIFI-5557
> URL: https://issues.apache.org/jira/browse/NIFI-5557
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Endre Kovacs
>Assignee: Endre Kovacs
>Priority: Major
>
> when using *PutHDFS* processor in a kerberized environment, with a flow
> "traffic" which approximately matches or less frequent then the lifetime of
> the ticket of the principal, we see this in the log:
> {code:java}
> INFO [Timer-Driven Process Thread-4] o.a.h.io.retry.RetryInvocationHandler
> Exception while invoking getFileInfo of class
> ClientNamenodeProtocolTranslatorPB over host2/ip2:8020 after 13 fail over
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for princi...@example.com to host2.example.com/ip2:8020;
> Host Details : local host is: "host1.example.com/ip1"; destination host is:
> "host2.example.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> at sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy135.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:254)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
> at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:222)
> {code}
> and the flowfile is routed to failure relationship.
> *To reproduce:*
> Create a principal in your KDC with two minutes ticket lifetime,
> and set up a similar flow:
> {code:java}
> GetFile => putHDFS - success- -> logAttributes
> \
> fail
>\
> -> logAttributes
> {code}
> copy a file to the input directory of the getFile processor. If the influx
> of the flowfile is much more frequent, then the expiration time of the ticket:
> {code:java}
> watch -n 5 "cp book.txt /path/to/input"
> {code}
> then the flow will successfully run without issue.
> If we adjust this, to:
> {code:java}
> watch -n 121 "cp book.txt /path/to/input"
> {code}
> then we will
[GitHub] nifi pull request #2971: NIFI-5557: handling expired ticket by rollback and ...
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2971#discussion_r219380028
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
---
@@ -266,6 +271,16 @@ public Object run() {
throw new
IOException(configuredRootDirPath.toString() + " could not be created");
}
changeOwner(context, hdfs, configuredRootDirPath,
flowFile);
+} catch (IOException e) {
+ boolean tgtExpired = hasCause(e, GSSException.class,
gsse -> GSSException.NO_CRED == gsse.getMajor());
+ if (tgtExpired) {
+getLogger().error(String.format("An error occured
while connecting to HDFS. Rolling back session, and penalizing flow file %s",
--- End diff --
The exception be logged here, in addition to the flowfile UUID. It might
be useful to have the stack trace and exception class available in the log, and
we shouldn't suppress/omit the actual GSSException from the logging.
It might also be a good idea to log this at the "warn" level, so that the
user can choose to not have these show as bulletins on the processor in the UI.
Since the flowfile is being rolled back, and hadoop-client will implicitly
acquire a new ticket, I don't think this should show as an error. @mcgilman,
@bbende, do either of you have a preference here?
---
[GitHub] nifi pull request #2971: NIFI-5557: handling expired ticket by rollback and ...
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2971#discussion_r219377632
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
---
@@ -266,6 +271,16 @@ public Object run() {
throw new
IOException(configuredRootDirPath.toString() + " could not be created");
}
changeOwner(context, hdfs, configuredRootDirPath,
flowFile);
+} catch (IOException e) {
--- End diff --
Thanks for changing this to use GSSException.getMajor(). I haven't tested
a ticket expiration occurring during the execution of a call to ugi.doAs (as
opposed to the ticket having expired before ugi.doAs is invoked), but I think
it would be a good idea to move this catch block to the top level try/catch
block of the PrivelegedExceptionAction passed to ugi.doAs().
---
[jira] [Created] (NIFI-5620) Standardize TLS Toolkit standalone and client/server command-line flags
Andy LoPresto created NIFI-5620: --- Summary: Standardize TLS Toolkit standalone and client/server command-line flags Key: NIFI-5620 URL: https://issues.apache.org/jira/browse/NIFI-5620 Project: Apache NiFi Issue Type: Improvement Components: Tools and Build Affects Versions: 1.7.1 Reporter: Andy LoPresto Assignee: Andy LoPresto There are some options which are available in standalone mode that are not available in client/server mode (i.e. setting explicit passwords for keys, etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-200) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFIREG-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622922#comment-16622922 ] ASF GitHub Bot commented on NIFIREG-200: Github user kevdoran commented on the issue: https://github.com/apache/nifi-registry/pull/142 Good catch @alopresto. I was simply trying to pin a single version of Guava as we were using 18.0 but 17.0 was being pulled in transitively. But I don't see any reason not to pin version 26.0, so I updated my branch to use that. Compilation and runtime seem unaffected. Thanks! > Upgrade version of Jetty > > > Key: NIFIREG-200 > URL: https://issues.apache.org/jira/browse/NIFIREG-200 > Project: NiFi Registry > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Andy LoPresto >Assignee: Kevin Doran >Priority: Blocker > Labels: jetty > Fix For: 0.3.0 > > > Spoke with Kevin off-list; he will make this change. > Please upgrade Jetty to version 9.4.11.x. > See [NIFI-5479|https://issues.apache.org/jira/browse/NIFI-5479]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry issue #142: NIFIREG-200 Update dependencies
Github user kevdoran commented on the issue: https://github.com/apache/nifi-registry/pull/142 Good catch @alopresto. I was simply trying to pin a single version of Guava as we were using 18.0 but 17.0 was being pulled in transitively. But I don't see any reason not to pin version 26.0, so I updated my branch to use that. Compilation and runtime seem unaffected. Thanks! ---
[jira] [Commented] (NIFIREG-200) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFIREG-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622909#comment-16622909 ] ASF GitHub Bot commented on NIFIREG-200: Github user alopresto commented on the issue: https://github.com/apache/nifi-registry/pull/142 @kevdoran any reason you decided on 18.0 for guava? Looks like the most recent release is 26.0 [Guava](https://github.com/google/guava). > Upgrade version of Jetty > > > Key: NIFIREG-200 > URL: https://issues.apache.org/jira/browse/NIFIREG-200 > Project: NiFi Registry > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Andy LoPresto >Assignee: Kevin Doran >Priority: Blocker > Labels: jetty > Fix For: 0.3.0 > > > Spoke with Kevin off-list; he will make this change. > Please upgrade Jetty to version 9.4.11.x. > See [NIFI-5479|https://issues.apache.org/jira/browse/NIFI-5479]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry issue #142: NIFIREG-200 Update dependencies
Github user alopresto commented on the issue: https://github.com/apache/nifi-registry/pull/142 @kevdoran any reason you decided on 18.0 for guava? Looks like the most recent release is 26.0 [Guava](https://github.com/google/guava). ---
[jira] [Commented] (NIFIREG-200) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFIREG-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622903#comment-16622903 ] ASF GitHub Bot commented on NIFIREG-200: Github user kevdoran commented on the issue: https://github.com/apache/nifi-registry/pull/142 @bbende if you could review this when you get a chance that would be appreciated > Upgrade version of Jetty > > > Key: NIFIREG-200 > URL: https://issues.apache.org/jira/browse/NIFIREG-200 > Project: NiFi Registry > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Andy LoPresto >Assignee: Kevin Doran >Priority: Blocker > Labels: jetty > Fix For: 0.3.0 > > > Spoke with Kevin off-list; he will make this change. > Please upgrade Jetty to version 9.4.11.x. > See [NIFI-5479|https://issues.apache.org/jira/browse/NIFI-5479]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-200) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFIREG-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622902#comment-16622902 ] ASF GitHub Bot commented on NIFIREG-200: GitHub user kevdoran opened a pull request: https://github.com/apache/nifi-registry/pull/142 NIFIREG-200 Update dependencies - Update Jetty to version 9.4.11.v20180605 - Update Spring Boot to version 2.0.4.RELEASE - Update Spring Security to version 5.0.7.RELEASE - Update Jackson to version 2.9.6 - Fix guava to version 18.0 (mutliple versions were being pulled in previously) You can merge this pull request into a Git repository by running: $ git pull https://github.com/kevdoran/nifi-registry NIFIREG-200-dep-version-bump Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-registry/pull/142.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #142 commit 66a9e07c8d2166e379eebc358b03bcf960700eee Author: Kevin Doran Date: 2018-09-21T00:01:49Z NIFIREG-200 Update dependencies - Update Jetty to version 9.4.11.v20180605 - Update Spring Boot to version 2.0.4.RELEASE - Update Spring Security to version 5.0.7.RELEASE - Update Jackson to version 2.9.6 - Fix guava to version 18.0 (mutliple versions were being pulled in previously) > Upgrade version of Jetty > > > Key: NIFIREG-200 > URL: https://issues.apache.org/jira/browse/NIFIREG-200 > Project: NiFi Registry > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Andy LoPresto >Assignee: Kevin Doran >Priority: Blocker > Labels: jetty > Fix For: 0.3.0 > > > Spoke with Kevin off-list; he will make this change. > Please upgrade Jetty to version 9.4.11.x. > See [NIFI-5479|https://issues.apache.org/jira/browse/NIFI-5479]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry issue #142: NIFIREG-200 Update dependencies
Github user kevdoran commented on the issue: https://github.com/apache/nifi-registry/pull/142 @bbende if you could review this when you get a chance that would be appreciated ---
[GitHub] nifi-registry pull request #142: NIFIREG-200 Update dependencies
GitHub user kevdoran opened a pull request: https://github.com/apache/nifi-registry/pull/142 NIFIREG-200 Update dependencies - Update Jetty to version 9.4.11.v20180605 - Update Spring Boot to version 2.0.4.RELEASE - Update Spring Security to version 5.0.7.RELEASE - Update Jackson to version 2.9.6 - Fix guava to version 18.0 (mutliple versions were being pulled in previously) You can merge this pull request into a Git repository by running: $ git pull https://github.com/kevdoran/nifi-registry NIFIREG-200-dep-version-bump Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-registry/pull/142.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #142 commit 66a9e07c8d2166e379eebc358b03bcf960700eee Author: Kevin Doran Date: 2018-09-21T00:01:49Z NIFIREG-200 Update dependencies - Update Jetty to version 9.4.11.v20180605 - Update Spring Boot to version 2.0.4.RELEASE - Update Spring Security to version 5.0.7.RELEASE - Update Jackson to version 2.9.6 - Fix guava to version 18.0 (mutliple versions were being pulled in previously) ---
[jira] [Commented] (NIFI-5514) MergeRecord does not create a merged FlowFile until a maximum threshold is reached
[ https://issues.apache.org/jira/browse/NIFI-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622815#comment-16622815 ] ASF GitHub Bot commented on NIFI-5514: -- Github user bdesert commented on the issue: https://github.com/apache/nifi/pull/2954 Reviewing... > MergeRecord does not create a merged FlowFile until a maximum threshold is > reached > -- > > Key: NIFI-5514 > URL: https://issues.apache.org/jira/browse/NIFI-5514 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.8.0 > > > MergeRecord allows the user to specify a minimum number of records. However, > if the minimum number of records is reached, the merged FlowFile is not > created unless the maximum number of records or max number of bytes is also > reached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2954: NIFI-5514: Fixed bugs in MergeRecord around minimum thresh...
Github user bdesert commented on the issue: https://github.com/apache/nifi/pull/2954 Reviewing... ---
[jira] [Updated] (NIFI-5619) Node can join the cluster with a processor running when cluster state for that processor is stopped
[ https://issues.apache.org/jira/browse/NIFI-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Storck updated NIFI-5619: -- Summary: Node can join the cluster with a processor running when cluster state for that processor is stopped (was: Node can join the cluster with processors running when cluster state for that processor is stopped) > Node can join the cluster with a processor running when cluster state for > that processor is stopped > --- > > Key: NIFI-5619 > URL: https://issues.apache.org/jira/browse/NIFI-5619 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Jeff Storck >Priority: Major > Attachments: Screen Shot 2018-09-20 at 4.53.44 PM.png, Screen Shot > 2018-09-20 at 4.54.07 PM.png, Screen Shot 2018-09-20 at 4.54.15 PM.png > > > When the cluster state for a processor is stopped, it is possible for a node > to join the cluster with that particular processor running. The UI will show > the processor as stopped, but going to the cluster summary and viewing the > node-specific status of that processor will show it's running on the node > that joined. > To reproduce this: > # Build the Node Decommissioning [PR > 3010|https://github.com/apache/nifi/pull/3010] > # Create a two-node cluster, does not need to be secure > # Create a flow with a GenerateFlowFile processor connected to an > UpdateAttribute processor > # Start the GenerateFlowFile processor to let flowfiles queue in the > connection to UpdateAttribute > # Disconnect the second node from the cluster > # Decommission the second node from the cluster > # Restart the node that was decommissioned from the cluster > # Once the node is reconnected, in the UI the GenerateFlowFile processor will > look like it is stopped > # Go to the Cluster Summary UI and click on the View Processor Details icon > which will show the processor still running on the node that was reconnected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5619) Node can join the cluster with processors running when cluster state for that processor is stopped
[ https://issues.apache.org/jira/browse/NIFI-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Storck updated NIFI-5619: -- Affects Version/s: 1.8.0 > Node can join the cluster with processors running when cluster state for that > processor is stopped > -- > > Key: NIFI-5619 > URL: https://issues.apache.org/jira/browse/NIFI-5619 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Jeff Storck >Priority: Major > Attachments: Screen Shot 2018-09-20 at 4.53.44 PM.png, Screen Shot > 2018-09-20 at 4.54.07 PM.png, Screen Shot 2018-09-20 at 4.54.15 PM.png > > > When the cluster state for a processor is stopped, it is possible for a node > to join the cluster with that particular processor running. The UI will show > the processor as stopped, but going to the cluster summary and viewing the > node-specific status of that processor will show it's running on the node > that joined. > To reproduce this: > # Build the Node Decommissioning [PR > 3010|https://github.com/apache/nifi/pull/3010] > # Create a two-node cluster, does not need to be secure > # Create a flow with a GenerateFlowFile processor connected to an > UpdateAttribute processor > # Start the GenerateFlowFile processor to let flowfiles queue in the > connection to UpdateAttribute > # Disconnect the second node from the cluster > # Decommission the second node from the cluster > # Restart the node that was decommissioned from the cluster > # Once the node is reconnected, in the UI the GenerateFlowFile processor will > look like it is stopped > # Go to the Cluster Summary UI and click on the View Processor Details icon > which will show the processor still running on the node that was reconnected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622706#comment-16622706
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user jtstorck commented on the issue:
https://github.com/apache/nifi/pull/3010
While testing the reconnecting of a decommissioned node to the cluster, the
issue detailed in [NIFI-5619](https://issues.apache.org/jira/browse/NIFI-5619)
was encountered.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3010: [WIP] NIFI-5585
Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/3010 While testing the reconnecting of a decommissioned node to the cluster, the issue detailed in [NIFI-5619](https://issues.apache.org/jira/browse/NIFI-5619) was encountered. ---
[jira] [Created] (NIFI-5619) Node can join the cluster with processors running when cluster state for that processor is stopped
Jeff Storck created NIFI-5619: - Summary: Node can join the cluster with processors running when cluster state for that processor is stopped Key: NIFI-5619 URL: https://issues.apache.org/jira/browse/NIFI-5619 Project: Apache NiFi Issue Type: Bug Reporter: Jeff Storck Attachments: Screen Shot 2018-09-20 at 4.53.44 PM.png, Screen Shot 2018-09-20 at 4.54.07 PM.png, Screen Shot 2018-09-20 at 4.54.15 PM.png When the cluster state for a processor is stopped, it is possible for a node to join the cluster with that particular processor running. The UI will show the processor as stopped, but going to the cluster summary and viewing the node-specific status of that processor will show it's running on the node that joined. To reproduce this: # Build the Node Decommissioning [PR 3010|https://github.com/apache/nifi/pull/3010] # Create a two-node cluster, does not need to be secure # Create a flow with a GenerateFlowFile processor connected to an UpdateAttribute processor # Start the GenerateFlowFile processor to let flowfiles queue in the connection to UpdateAttribute # Disconnect the second node from the cluster # Decommission the second node from the cluster # Restart the node that was decommissioned from the cluster # Once the node is reconnected, in the UI the GenerateFlowFile processor will look like it is stopped # Go to the Cluster Summary UI and click on the View Processor Details icon which will show the processor still running on the node that was reconnected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622642#comment-16622642
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219296344
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/SocketLoadBalancedFlowFileQueue.java
---
@@ -204,6 +206,19 @@ public synchronized void setLoadBalanceStrategy(final
LoadBalanceStrategy strate
setFlowFilePartitioner(partitioner);
}
+@Override
+public void decommissionQueue() {
+if (clusterCoordinator == null) {
+// Not clustered, so don't change partitions
+return;
+}
+
+// TODO set decommissioned boolean
--- End diff --
I think the TODO can go
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622633#comment-16622633
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219293897
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java
---
@@ -821,7 +878,9 @@ void notifyOthersOfNodeStatusChange(final
NodeConnectionStatus updatedStatus, fi
// Otherwise, get the active coordinator (or wait for one to
become active) and then notify the coordinator.
final Set nodesToNotify;
if (notifyAllNodes) {
-nodesToNotify =
getNodeIdentifiers(NodeConnectionState.CONNECTED,
NodeConnectionState.CONNECTING);
+// TODO notify all nodes
--- End diff --
I think we can remove this TODO and the line below it.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622637#comment-16622637
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219297604
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/NonLocalPartitionPartitioner.java
---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * PReturns remote partitions when queried for a partition; never returns
the {@link LocalQueuePartition}.
+ */
+public class NonLocalPartitionPartitioner implements FlowFilePartitioner {
+private final AtomicLong counter = new AtomicLong(0L);
+
+@Override
+public QueuePartition getPartition(final FlowFileRecord flowFile,
final QueuePartition[] partitions, final QueuePartition localPartition) {
+QueuePartition remotePartition = null;
+for (int i = 0, numPartitions = partitions.length; i <
numPartitions; i++) {
+final long count = counter.getAndIncrement();
--- End diff --
I think the logic here is just a bit off, because if we have say 5
partitions but many threads, it could be the case that the first iteration gets
count = 3 and that's the local partition. The next iteration could get count =
8 so index = 3, which is the local partition. The next iteration gets count =
13 so index = 3, then count = 18, then count = 23. At this point we'd have
pointed to the local partition 5 times in a row and thrown
IllegalStateException.
Instead, I think we should do something like:
{code}
final long startIndex = counter.getAndIncrement();
for (int i=0; i < partitions.length; i++) {
final int index = (int) (startIndex + i) % partitions.length;
final QueuePartition partition = partitions[index];
...
}
{code}
This way, we still get the round-robin-ing effect but ensure that we try
all partitions before throwing an IllegalStateException.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622631#comment-16622631
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219293820
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java
---
@@ -526,6 +579,10 @@ public void removeNode(final NodeIdentifier nodeId,
final String userDn) {
storeState();
}
+private void onNodeDecommissioned(final NodeIdentifier nodeId) {
+eventListeners.stream().forEach(listener ->
listener.onNodeDecommissioned(nodeId));
+}
+
private void onNodeRemoved(final NodeIdentifier nodeId) {
eventListeners.stream().forEach(listener ->
listener.onNodeRemoved(nodeId));
--- End diff --
Even though you didn't add this line, I see that it can also have the call
to stream() removed :)
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622640#comment-16622640
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219294892
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java
---
@@ -662,6 +682,39 @@ private void handleReconnectionRequest(final
ReconnectionRequestMessage request)
}
}
+private void handleDecommissionRequest(final DecommissionMessage
request) throws InterruptedException {
+logger.info("Received decommission request message from manager
with explanation: " + request.getExplanation());
--- End diff --
I think this should say "from cluster coordinator" rather than "from
manager"
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622635#comment-16622635
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219293725
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java
---
@@ -526,6 +579,10 @@ public void removeNode(final NodeIdentifier nodeId,
final String userDn) {
storeState();
}
+private void onNodeDecommissioned(final NodeIdentifier nodeId) {
+eventListeners.stream().forEach(listener ->
listener.onNodeDecommissioned(nodeId));
--- End diff --
Can get rid of the call to stream() here and just call forEach() on
eventListeners.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622641#comment-16622641
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219296392
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/SocketLoadBalancedFlowFileQueue.java
---
@@ -534,6 +549,11 @@ public void setNodeIdentifiers(final
Set updatedNodeIdentifiers,
return;
}
+// TODO check if queue is already decommissioned
--- End diff --
Can get rid of the TODO
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622630#comment-16622630
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219293091
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java
---
@@ -180,6 +181,15 @@ public AsyncClusterResponse replicate(NiFiUser user,
String method, URI uri, Obj
}
}
+final List decommissioning =
stateMap.get(NodeConnectionState.DECOMMISSIONING);
--- End diff --
I think we need to also check for any nodes that are decommissioned as
well. At first i thought maybe we should just allow it if a node is
decommissioned. But then it's a little weird because the UI still will show 3/4
nodes connected (for example). And in all other cases, we would reject the
request because the cluster is not at 100% connected. So I think we need to
disallow even for decommissioned nodes.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622638#comment-16622638
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219296555
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/NonLocalPartitionPartitioner.java
---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * PReturns remote partitions when queried for a partition; never returns
the {@link LocalQueuePartition}.
--- End diff --
Typo: "PReturns"
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622636#comment-16622636
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219294399
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/heartbeat/TestAbstractHeartbeatMonitor.java
---
@@ -244,11 +245,26 @@ public synchronized void
finishNodeConnection(NodeIdentifier nodeId) {
statuses.put(nodeId, new NodeConnectionStatus(nodeId,
NodeConnectionState.CONNECTED));
}
+@Override
+public synchronized void finishNodeDecommission(NodeIdentifier
nodeId) {
+statuses.put(nodeId, new NodeConnectionStatus(nodeId,
NodeConnectionState.DECOMMISSIONED));
+}
+
+@Override
+public synchronized void requestNodeDecommission(NodeIdentifier
nodeId, DecommissionCode decommissionCode, String explanation) {
+statuses.put(nodeId, new NodeConnectionStatus(nodeId,
NodeConnectionState.DECOMMISSIONED));
+}
+
@Override
public synchronized void requestNodeDisconnect(NodeIdentifier
nodeId, DisconnectionCode disconnectionCode, String explanation) {
statuses.put(nodeId, new NodeConnectionStatus(nodeId,
NodeConnectionState.DISCONNECTED));
}
+//@Override
--- End diff --
Can remove.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622634#comment-16622634
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219295502
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java
---
@@ -662,6 +682,39 @@ private void handleReconnectionRequest(final
ReconnectionRequestMessage request)
}
}
+private void handleDecommissionRequest(final DecommissionMessage
request) throws InterruptedException {
+logger.info("Received decommission request message from manager
with explanation: " + request.getExplanation());
+decommission(request.getExplanation());
+}
+
+private void decommission(final String explanation) throws
InterruptedException {
+writeLock.lock();
+try {
+
+logger.info("Decommissioning node due to " + explanation);
+
+// mark node as decommissioning
+controller.setConnectionStatus(new
NodeConnectionStatus(nodeId, NodeConnectionState.DECOMMISSIONING,
DecommissionCode.DECOMMISSIONED, explanation));
+// request to stop all processors on node
+controller.stopAllProcessors();
--- End diff --
In addition to calling stopAllProcessors() I think we should be terminating
all active processors as well. We can do this by getting the Root Process Group
from the FlowController, then from that getting all Processors and for each one
if processor.getScheduledState() == ScheduledState.STOPPED (do this in case a
processor is disabled), call ProcessGroup.terminateProcessor(ProcessorNode) and
then recursing through all groups.
This will ensure that even if a Processor has hold of a FlowFile and makes
no progress we can still push the FlowFiles out to other nodes in the cluster
in order to decommission.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622629#comment-16622629
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219271614
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster-protocol/src/main/java/org/apache/nifi/cluster/coordination/ClusterCoordinator.java
---
@@ -72,6 +91,16 @@
*/
void requestNodeDisconnect(NodeIdentifier nodeId, DisconnectionCode
disconnectionCode, String explanation);
+///**
--- End diff --
We can get rid of this code that's commented out.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622639#comment-16622639
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219293350
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java
---
@@ -494,6 +539,14 @@ public void requestNodeDisconnect(final NodeIdentifier
nodeId, final Disconnecti
disconnectAsynchronously(request, 10, 5);
}
+//@Override
--- End diff --
We can remove this.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622632#comment-16622632
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219294246
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java
---
@@ -841,6 +900,34 @@ void notifyOthersOfNodeStatusChange(final
NodeConnectionStatus updatedStatus, fi
senderListener.notifyNodeStatusChange(nodesToNotify, message);
}
+private void decommissionAsynchronously(final DecommissionMessage
request, final int attempts, final int retrySeconds) {
+final Thread decommissionThread = new Thread(new Runnable() {
+@Override
+public void run() {
+final NodeIdentifier nodeId = request.getNodeId();
+
+for (int i = 0; i < attempts; i++) {
+try {
+senderListener.decommission(request);
+reportEvent(nodeId, Severity.INFO, "Node was
decommissioned due to " + request.getExplanation());
+return;
+} catch (final Exception e) {
+logger.error("Failed to notify {} that it has been
decommissioned from the cluster due to {}", request.getNodeId(),
request.getExplanation());
--- End diff --
We should probably log the stack trace here as well.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request: https://github.com/apache/nifi/pull/3010#discussion_r219296392 --- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/SocketLoadBalancedFlowFileQueue.java --- @@ -534,6 +549,11 @@ public void setNodeIdentifiers(final Set updatedNodeIdentifiers, return; } +// TODO check if queue is already decommissioned --- End diff -- Can get rid of the TODO ---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219294892
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java
---
@@ -662,6 +682,39 @@ private void handleReconnectionRequest(final
ReconnectionRequestMessage request)
}
}
+private void handleDecommissionRequest(final DecommissionMessage
request) throws InterruptedException {
+logger.info("Received decommission request message from manager
with explanation: " + request.getExplanation());
--- End diff --
I think this should say "from cluster coordinator" rather than "from
manager"
---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219293725
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java
---
@@ -526,6 +579,10 @@ public void removeNode(final NodeIdentifier nodeId,
final String userDn) {
storeState();
}
+private void onNodeDecommissioned(final NodeIdentifier nodeId) {
+eventListeners.stream().forEach(listener ->
listener.onNodeDecommissioned(nodeId));
--- End diff --
Can get rid of the call to stream() here and just call forEach() on
eventListeners.
---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219293820
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java
---
@@ -526,6 +579,10 @@ public void removeNode(final NodeIdentifier nodeId,
final String userDn) {
storeState();
}
+private void onNodeDecommissioned(final NodeIdentifier nodeId) {
+eventListeners.stream().forEach(listener ->
listener.onNodeDecommissioned(nodeId));
+}
+
private void onNodeRemoved(final NodeIdentifier nodeId) {
eventListeners.stream().forEach(listener ->
listener.onNodeRemoved(nodeId));
--- End diff --
Even though you didn't add this line, I see that it can also have the call
to stream() removed :)
---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219296555
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/NonLocalPartitionPartitioner.java
---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * PReturns remote partitions when queried for a partition; never returns
the {@link LocalQueuePartition}.
--- End diff --
Typo: "PReturns"
---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request: https://github.com/apache/nifi/pull/3010#discussion_r219271614 --- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster-protocol/src/main/java/org/apache/nifi/cluster/coordination/ClusterCoordinator.java --- @@ -72,6 +91,16 @@ */ void requestNodeDisconnect(NodeIdentifier nodeId, DisconnectionCode disconnectionCode, String explanation); +///** --- End diff -- We can get rid of this code that's commented out. ---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219294246
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java
---
@@ -841,6 +900,34 @@ void notifyOthersOfNodeStatusChange(final
NodeConnectionStatus updatedStatus, fi
senderListener.notifyNodeStatusChange(nodesToNotify, message);
}
+private void decommissionAsynchronously(final DecommissionMessage
request, final int attempts, final int retrySeconds) {
+final Thread decommissionThread = new Thread(new Runnable() {
+@Override
+public void run() {
+final NodeIdentifier nodeId = request.getNodeId();
+
+for (int i = 0; i < attempts; i++) {
+try {
+senderListener.decommission(request);
+reportEvent(nodeId, Severity.INFO, "Node was
decommissioned due to " + request.getExplanation());
+return;
+} catch (final Exception e) {
+logger.error("Failed to notify {} that it has been
decommissioned from the cluster due to {}", request.getNodeId(),
request.getExplanation());
--- End diff --
We should probably log the stack trace here as well.
---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219294399
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/heartbeat/TestAbstractHeartbeatMonitor.java
---
@@ -244,11 +245,26 @@ public synchronized void
finishNodeConnection(NodeIdentifier nodeId) {
statuses.put(nodeId, new NodeConnectionStatus(nodeId,
NodeConnectionState.CONNECTED));
}
+@Override
+public synchronized void finishNodeDecommission(NodeIdentifier
nodeId) {
+statuses.put(nodeId, new NodeConnectionStatus(nodeId,
NodeConnectionState.DECOMMISSIONED));
+}
+
+@Override
+public synchronized void requestNodeDecommission(NodeIdentifier
nodeId, DecommissionCode decommissionCode, String explanation) {
+statuses.put(nodeId, new NodeConnectionStatus(nodeId,
NodeConnectionState.DECOMMISSIONED));
+}
+
@Override
public synchronized void requestNodeDisconnect(NodeIdentifier
nodeId, DisconnectionCode disconnectionCode, String explanation) {
statuses.put(nodeId, new NodeConnectionStatus(nodeId,
NodeConnectionState.DISCONNECTED));
}
+//@Override
--- End diff --
Can remove.
---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219295502
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java
---
@@ -662,6 +682,39 @@ private void handleReconnectionRequest(final
ReconnectionRequestMessage request)
}
}
+private void handleDecommissionRequest(final DecommissionMessage
request) throws InterruptedException {
+logger.info("Received decommission request message from manager
with explanation: " + request.getExplanation());
+decommission(request.getExplanation());
+}
+
+private void decommission(final String explanation) throws
InterruptedException {
+writeLock.lock();
+try {
+
+logger.info("Decommissioning node due to " + explanation);
+
+// mark node as decommissioning
+controller.setConnectionStatus(new
NodeConnectionStatus(nodeId, NodeConnectionState.DECOMMISSIONING,
DecommissionCode.DECOMMISSIONED, explanation));
+// request to stop all processors on node
+controller.stopAllProcessors();
--- End diff --
In addition to calling stopAllProcessors() I think we should be terminating
all active processors as well. We can do this by getting the Root Process Group
from the FlowController, then from that getting all Processors and for each one
if processor.getScheduledState() == ScheduledState.STOPPED (do this in case a
processor is disabled), call ProcessGroup.terminateProcessor(ProcessorNode) and
then recursing through all groups.
This will ensure that even if a Processor has hold of a FlowFile and makes
no progress we can still push the FlowFiles out to other nodes in the cluster
in order to decommission.
---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219296344
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/SocketLoadBalancedFlowFileQueue.java
---
@@ -204,6 +206,19 @@ public synchronized void setLoadBalanceStrategy(final
LoadBalanceStrategy strate
setFlowFilePartitioner(partitioner);
}
+@Override
+public void decommissionQueue() {
+if (clusterCoordinator == null) {
+// Not clustered, so don't change partitions
+return;
+}
+
+// TODO set decommissioned boolean
--- End diff --
I think the TODO can go
---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219293897
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java
---
@@ -821,7 +878,9 @@ void notifyOthersOfNodeStatusChange(final
NodeConnectionStatus updatedStatus, fi
// Otherwise, get the active coordinator (or wait for one to
become active) and then notify the coordinator.
final Set nodesToNotify;
if (notifyAllNodes) {
-nodesToNotify =
getNodeIdentifiers(NodeConnectionState.CONNECTED,
NodeConnectionState.CONNECTING);
+// TODO notify all nodes
--- End diff --
I think we can remove this TODO and the line below it.
---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request: https://github.com/apache/nifi/pull/3010#discussion_r219293091 --- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java --- @@ -180,6 +181,15 @@ public AsyncClusterResponse replicate(NiFiUser user, String method, URI uri, Obj } } +final List decommissioning = stateMap.get(NodeConnectionState.DECOMMISSIONING); --- End diff -- I think we need to also check for any nodes that are decommissioned as well. At first i thought maybe we should just allow it if a node is decommissioned. But then it's a little weird because the UI still will show 3/4 nodes connected (for example). And in all other cases, we would reject the request because the cluster is not at 100% connected. So I think we need to disallow even for decommissioned nodes. ---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3010#discussion_r219297604
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/NonLocalPartitionPartitioner.java
---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * PReturns remote partitions when queried for a partition; never returns
the {@link LocalQueuePartition}.
+ */
+public class NonLocalPartitionPartitioner implements FlowFilePartitioner {
+private final AtomicLong counter = new AtomicLong(0L);
+
+@Override
+public QueuePartition getPartition(final FlowFileRecord flowFile,
final QueuePartition[] partitions, final QueuePartition localPartition) {
+QueuePartition remotePartition = null;
+for (int i = 0, numPartitions = partitions.length; i <
numPartitions; i++) {
+final long count = counter.getAndIncrement();
--- End diff --
I think the logic here is just a bit off, because if we have say 5
partitions but many threads, it could be the case that the first iteration gets
count = 3 and that's the local partition. The next iteration could get count =
8 so index = 3, which is the local partition. The next iteration gets count =
13 so index = 3, then count = 18, then count = 23. At this point we'd have
pointed to the local partition 5 times in a row and thrown
IllegalStateException.
Instead, I think we should do something like:
{code}
final long startIndex = counter.getAndIncrement();
for (int i=0; i < partitions.length; i++) {
final int index = (int) (startIndex + i) % partitions.length;
final QueuePartition partition = partitions[index];
...
}
{code}
This way, we still get the round-robin-ing effect but ensure that we try
all partitions before throwing an IllegalStateException.
---
[GitHub] nifi pull request #3010: [WIP] NIFI-5585
Github user markap14 commented on a diff in the pull request: https://github.com/apache/nifi/pull/3010#discussion_r219293350 --- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/node/NodeClusterCoordinator.java --- @@ -494,6 +539,14 @@ public void requestNodeDisconnect(final NodeIdentifier nodeId, final Disconnecti disconnectAsynchronously(request, 10, 5); } +//@Override --- End diff -- We can remove this. ---
[jira] [Commented] (MINIFICPP-614) Reduce overhead of AgentInformation::serialize
[ https://issues.apache.org/jira/browse/MINIFICPP-614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622621#comment-16622621 ] ASF GitHub Bot commented on MINIFICPP-614: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/401 > Reduce overhead of AgentInformation::serialize > -- > > Key: MINIFICPP-614 > URL: https://issues.apache.org/jira/browse/MINIFICPP-614 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Serialize is taking more cycles than necessary. we can improve this > dramatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-614) Reduce overhead of AgentInformation::serialize
[ https://issues.apache.org/jira/browse/MINIFICPP-614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-614: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Reduce overhead of AgentInformation::serialize > -- > > Key: MINIFICPP-614 > URL: https://issues.apache.org/jira/browse/MINIFICPP-614 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Serialize is taking more cycles than necessary. we can improve this > dramatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #401: MINIFICPP-614: Don't rebuild agent inform...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/401 ---
[jira] [Updated] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Storck updated NIFI-5585:
--
Description:
Allow a node in the cluster to be decommissioned, rebalancing flowfiles on the
node to be decommissioned to the other active nodes. This work depends on
NIFI-5516.
Similar to the client sending PUT request a DISCONNECTING message to
cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request to
the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
DECOMMISSIONING request will be idempotent.
The steps to decommission a node and remove it from the cluster are:
# Send request to disconnect the node
# Once disconnect completes, send request to decommission the node.
# Once decommission completes, send request to delete node.
When an error occurs and the node can not complete decommissioning, the user
can:
# Send request to delete the node from the cluster
# Diagnose why the node had issues with the decommission (out of memory, no
network connection, etc) and address the issue
# Restart NiFi on the node to so that it will reconnect to the cluster
# Go through the steps to decommission and remove a node
Toolkit CLI commands for retrieving a list of nodes and
disconnecting/decommissioning/deleting nodes have been added.
was:
Allow a node in the cluster to be decommissioned, rebalancing flowfiles on the
node to be decommissioned to the other active nodes. This work depends on
NIFI-5516.
Similar to the client sending PUT request a DISCONNECTING message to
cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request to
the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
DECOMMISSIONING request will be idempotent.
The steps to decommission a node and remove it from the cluster are:
# Send request to disconnect the node
# Once disconnect completes, send request to decommission the node.
# Once decommission completes, send request to delete node.
When an error occurs and the node can not complete decommissioning, the user
can:
# Send request to delete the node from the cluster
# Diagnose why the node had issues with the decommission (out of memory, no
network connection, etc) and address the issue
# Restart NiFi on the node to so that it will reconnect to the cluster
# Go through the steps to decommission and remove a node
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
> Toolkit CLI commands for retrieving a list of nodes and
> disconnecting/decommissioning/deleting nodes have been added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Storck updated NIFI-5585:
--
Comment: was deleted
(was: WIP Apache NiFi [PR 3010|https://github.com/apache/nifi/pull/3010])
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Storck updated NIFI-5585:
--
Status: Patch Available (was: In Progress)
WIP Apache NiFi [PR 3010|https://github.com/apache/nifi/pull/3010]
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFIREG-186) Create Authorizer implementation that uses Apache Ranger
[ https://issues.apache.org/jira/browse/NIFIREG-186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Doran resolved NIFIREG-186. - Resolution: Fixed > Create Authorizer implementation that uses Apache Ranger > > > Key: NIFIREG-186 > URL: https://issues.apache.org/jira/browse/NIFIREG-186 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > Fix For: 0.3.0 > > > In addition to the standard file-based Authorizer, we should provide an > Authorizer implementation that uses Apache Ranger, so that users implement > centralized authorization against both NiFi and NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-186) Create Authorizer implementation that uses Apache Ranger
[ https://issues.apache.org/jira/browse/NIFIREG-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622584#comment-16622584 ] ASF GitHub Bot commented on NIFIREG-186: Github user asfgit closed the pull request at: https://github.com/apache/nifi-registry/pull/141 > Create Authorizer implementation that uses Apache Ranger > > > Key: NIFIREG-186 > URL: https://issues.apache.org/jira/browse/NIFIREG-186 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > Fix For: 0.3.0 > > > In addition to the standard file-based Authorizer, we should provide an > Authorizer implementation that uses Apache Ranger, so that users implement > centralized authorization against both NiFi and NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry pull request #141: NIFIREG-186 - Referencing the correct paren...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-registry/pull/141 ---
[jira] [Commented] (NIFI-5585) Decommision Nodes from Cluster
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622580#comment-16622580
]
ASF GitHub Bot commented on NIFI-5585:
--
Github user jtstorck commented on the issue:
https://github.com/apache/nifi/pull/3010
While testing the reconnection of a decommissioned node, with ~24,000 files
split between the two nodes, an error occurred after reconnecting the
decomissioned node and attempting to drop all the flowfiles in the queue, from
both nodes:
```
2018-09-20 15:25:22,594 ERROR [Drop FlowFiles for Connection
cbbf2971-0165-1000--94b81269]
o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Failed to drop FlowFiles for
org.apache.nifi.controller.queue.clustered.SocketLoadBalancedFlowFileQueue@469caf69
java.lang.IllegalArgumentException: null
at org.apache.nifi.controller.queue.QueueSize.(QueueSize.java:31)
at org.apache.nifi.controller.queue.QueueSize.add(QueueSize.java:67)
at
org.apache.nifi.controller.queue.clustered.SocketLoadBalancedFlowFileQueue.adjustSize(SocketLoadBalancedFlowFileQueue.java:514)
at
org.apache.nifi.controller.queue.clustered.SocketLoadBalancedFlowFileQueue.dropFlowFiles(SocketLoadBalancedFlowFileQueue.java:903)
at
org.apache.nifi.controller.queue.AbstractFlowFileQueue$2.run(AbstractFlowFileQueue.java:285)
at java.lang.Thread.run(Thread.java:748)
```
The decommission operation was successful, all flowfiles were moved from
the decommissioned node to the other node. After reconnecting the
decommissioned node, I couldn't clear the flowfile queue.
After restarting the cluster (both nodes), the queue showed as empty.
> Decommision Nodes from Cluster
> --
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be decommissioned, rebalancing flowfiles on
> the node to be decommissioned to the other active nodes. This work depends
> on NIFI-5516.
> Similar to the client sending PUT request a DISCONNECTING message to
> cluster/nodes/\{id}, a DECOMMISSIONING message can be sent as a PUT request
> to the same URI to initiate a DECOMMISSION for a DISCONNECTED node. The
> DECOMMISSIONING request will be idempotent.
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to decommission the node.
> # Once decommission completes, send request to delete node.
> When an error occurs and the node can not complete decommissioning, the user
> can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the decommission (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission and remove a node
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3010: [WIP] NIFI-5585
Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/3010 While testing the reconnection of a decommissioned node, with ~24,000 files split between the two nodes, an error occurred after reconnecting the decomissioned node and attempting to drop all the flowfiles in the queue, from both nodes: ``` 2018-09-20 15:25:22,594 ERROR [Drop FlowFiles for Connection cbbf2971-0165-1000--94b81269] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Failed to drop FlowFiles for org.apache.nifi.controller.queue.clustered.SocketLoadBalancedFlowFileQueue@469caf69 java.lang.IllegalArgumentException: null at org.apache.nifi.controller.queue.QueueSize.(QueueSize.java:31) at org.apache.nifi.controller.queue.QueueSize.add(QueueSize.java:67) at org.apache.nifi.controller.queue.clustered.SocketLoadBalancedFlowFileQueue.adjustSize(SocketLoadBalancedFlowFileQueue.java:514) at org.apache.nifi.controller.queue.clustered.SocketLoadBalancedFlowFileQueue.dropFlowFiles(SocketLoadBalancedFlowFileQueue.java:903) at org.apache.nifi.controller.queue.AbstractFlowFileQueue$2.run(AbstractFlowFileQueue.java:285) at java.lang.Thread.run(Thread.java:748) ``` The decommission operation was successful, all flowfiles were moved from the decommissioned node to the other node. After reconnecting the decommissioned node, I couldn't clear the flowfile queue. After restarting the cluster (both nodes), the queue showed as empty. ---
[jira] [Created] (NIFI-5618) NullPointerException is thrown if attempting to view details of a Provenance Event on a node that is disconnected from cluster
Mark Payne created NIFI-5618:
Summary: NullPointerException is thrown if attempting to view
details of a Provenance Event on a node that is disconnected from cluster
Key: NIFI-5618
URL: https://issues.apache.org/jira/browse/NIFI-5618
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
I have a cluster of 2 nodes. I disconnected one of the nodes, then did a
Provenance Query. This returned the results correctly. However, when I tried to
view the details of the provenance event, I got an error in the UI indicating
that I should check my logs. User log has the following (partial) stack trace:
{code:java}
2018-09-20 15:16:36,049 ERROR [NiFi Web Server-177]
o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
java.lang.NullPointerException. Returning Internal Server Error response.
java.lang.NullPointerException: null
at
org.apache.nifi.web.api.ProvenanceEventResource.getProvenanceEvent(ProvenanceEventResource.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268){code}
It appears to be due to the fact that since the node is disconnected, the
clusterNodeId is not provided in the REST API call. So the following block of
code:
{code:java}
final ClusterCoordinator coordinator = getClusterCoordinator();
if (coordinator != null) {
final NodeIdentifier nodeId = coordinator.getNodeIdentifier(clusterNodeId);
event.setClusterNodeAddress(nodeId.getApiAddress() + ":" +
nodeId.getApiPort());
}{code}
results in calling coordinator.getNodeIdentifier(null), which returns null for
the nodeId. We then call nodeId.getApiAddress(), throwing a NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFIREG-196) Upgrade lodash, parsejson, https-proxy-agent
[ https://issues.apache.org/jira/browse/NIFIREG-196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman resolved NIFIREG-196. - Resolution: Fixed > Upgrade lodash, parsejson, https-proxy-agent > > > Key: NIFIREG-196 > URL: https://issues.apache.org/jira/browse/NIFIREG-196 > Project: NiFi Registry > Issue Type: Bug >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Fix For: 0.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #400: MINIFICPP-613: Remove undocumented option...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/400 ---
[jira] [Commented] (MINIFICPP-613) Remove rooturi local option
[ https://issues.apache.org/jira/browse/MINIFICPP-613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622552#comment-16622552 ] ASF GitHub Bot commented on MINIFICPP-613: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/400 > Remove rooturi local option > --- > > Key: MINIFICPP-613 > URL: https://issues.apache.org/jira/browse/MINIFICPP-613 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Remove an unused option for the listener root uri -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-613) Remove rooturi local option
[ https://issues.apache.org/jira/browse/MINIFICPP-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-613: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Remove rooturi local option > --- > > Key: MINIFICPP-613 > URL: https://issues.apache.org/jira/browse/MINIFICPP-613 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Remove an unused option for the listener root uri -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5617) Coordinator role change during message handling
Jeff Storck created NIFI-5617: - Summary: Coordinator role change during message handling Key: NIFI-5617 URL: https://issues.apache.org/jira/browse/NIFI-5617 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.7.1 Reporter: Jeff Storck A scenario may exist where the cluster's Coordinator role changes from one node to another, while the node that was previously the Coordinator is processing a message. This potential issue has not been seen/reported as of NiFi 1.7.1. Steps to create this scenario for testing will be added to this JIRA as they become available. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-186) Create Authorizer implementation that uses Apache Ranger
[ https://issues.apache.org/jira/browse/NIFIREG-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622541#comment-16622541 ] ASF GitHub Bot commented on NIFIREG-186: GitHub user mcgilman opened a pull request: https://github.com/apache/nifi-registry/pull/141 NIFIREG-186 - Referencing the correct parent artifact NIFIREG-186: - Referencing the correct parent artifact You can merge this pull request into a Git repository by running: $ git pull https://github.com/mcgilman/nifi-registry NIFIREG-186 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-registry/pull/141.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #141 commit 72bec8135fcab7f1cc9c704d480996afa2bb8e45 Author: Matt Gilman Date: 2018-09-20T19:05:01Z NIFIREG-186 Referencing the correct parent artifact. > Create Authorizer implementation that uses Apache Ranger > > > Key: NIFIREG-186 > URL: https://issues.apache.org/jira/browse/NIFIREG-186 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > Fix For: 0.3.0 > > > In addition to the standard file-based Authorizer, we should provide an > Authorizer implementation that uses Apache Ranger, so that users implement > centralized authorization against both NiFi and NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry pull request #141: NIFIREG-186 - Referencing the correct paren...
GitHub user mcgilman opened a pull request: https://github.com/apache/nifi-registry/pull/141 NIFIREG-186 - Referencing the correct parent artifact NIFIREG-186: - Referencing the correct parent artifact You can merge this pull request into a Git repository by running: $ git pull https://github.com/mcgilman/nifi-registry NIFIREG-186 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-registry/pull/141.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #141 commit 72bec8135fcab7f1cc9c704d480996afa2bb8e45 Author: Matt Gilman Date: 2018-09-20T19:05:01Z NIFIREG-186 Referencing the correct parent artifact. ---
[jira] [Updated] (MINIFICPP-610) Generate type descriptions
[ https://issues.apache.org/jira/browse/MINIFICPP-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-610: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Generate type descriptions > -- > > Key: MINIFICPP-610 > URL: https://issues.apache.org/jira/browse/MINIFICPP-610 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-610) Generate type descriptions
[ https://issues.apache.org/jira/browse/MINIFICPP-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622530#comment-16622530 ] ASF GitHub Bot commented on MINIFICPP-610: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/399 > Generate type descriptions > -- > > Key: MINIFICPP-610 > URL: https://issues.apache.org/jira/browse/MINIFICPP-610 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (NIFIREG-186) Create Authorizer implementation that uses Apache Ranger
[ https://issues.apache.org/jira/browse/NIFIREG-186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Doran reopened NIFIREG-186: - Reopening to fix maven pom files > Create Authorizer implementation that uses Apache Ranger > > > Key: NIFIREG-186 > URL: https://issues.apache.org/jira/browse/NIFIREG-186 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > Fix For: 0.3.0 > > > In addition to the standard file-based Authorizer, we should provide an > Authorizer implementation that uses Apache Ranger, so that users implement > centralized authorization against both NiFi and NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #399: MINIFICPP-610: Add description to types
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/399 ---
[jira] [Commented] (NIFI-4610) Hanging Processor: ExtractHL7Attributes
[
https://issues.apache.org/jira/browse/NIFI-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622526#comment-16622526
]
Mohammed Nadeem commented on NIFI-4610:
---
[~mattyb149] Should I do a quick validation check for null?
> Hanging Processor: ExtractHL7Attributes
> ---
>
> Key: NIFI-4610
> URL: https://issues.apache.org/jira/browse/NIFI-4610
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Affects Versions: 1.4.0
> Environment: Hanging Processor: ExtractHL7Attributes
> NiFi Version 1.4.0
> Tag nifi-1.4.0-RC2
> Build Date/Time 09/28/2017 14:58:26 CDT
> Name Windows Server 2012 R2
> Version 6.3
> Architecture x86
>Reporter: Eric Thompson
>Priority: Blocker
> Attachments: Display.png
>
>
> I am getting an error of 'Administratively Yielded for 1 sec due to
> processing failure' and the files that are triggering the error in the
> processor are not getting routed to the failure processor but seem to loop
> and stop up the first processor. Any ideas on how to resolve?
>
> I would expect the files that are loaded that cause the error would get
> routed, but instead it appears as though it eventually just stops accepting
> new files all together since the queue
> Sample Error in nifi-app.log:
> 2017-11-09 06:03:38,835 ERROR [Timer-Driven Process Thread-7]
> o.a.n.p.hl7.ExtractHL7Attributes
> ExtractHL7Attributes[id=015f105a-8bc0-1e9a-d683-34dd30c9d744]
> ExtractHL7Attributes[id=015f105a-8bc0-1e9a-d683-34dd30c9d744] failed to
> process session due to java.lang.NullPointerException: {}
> java.lang.NullPointerException: null
> at
> org.apache.nifi.processors.hl7.ExtractHL7Attributes.getAllFields(ExtractHL7Attributes.java:287)
> at
> org.apache.nifi.processors.hl7.ExtractHL7Attributes.getAttributes(ExtractHL7Attributes.java:217)
> at
> org.apache.nifi.processors.hl7.ExtractHL7Attributes.onTrigger(ExtractHL7Attributes.java:199)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> 2017-11-09 06:03:38,835 WARN [Timer-Driven Process Thread-7]
> o.a.n.p.hl7.ExtractHL7Attributes
> ExtractHL7Attributes[id=015f105a-8bc0-1e9a-d683-34dd30c9d744] Processor
> Administratively Yielded for 1 sec due to processing failure
> 2017-11-09 06:03:38,835 WARN [Timer-Driven Process Thread-7]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> ExtractHL7Attributes[id=015f105a-8bc0-1e9a-d683-34dd30c9d744] due to uncaught
> Exception: java.lang.NullPointerException
> 2017-11-09 06:03:38,835 WARN [Timer-Driven Process Thread-7]
> o.a.n.c.t.ContinuallyRunProcessorTask
> java.lang.NullPointerException: null
> at
> org.apache.nifi.processors.hl7.ExtractHL7Attributes.getAllFields(ExtractHL7Attributes.java:287)
> at
> org.apache.nifi.processors.hl7.ExtractHL7Attributes.getAttributes(ExtractHL7Attributes.java:217)
> at
> org.apache.nifi.processors.hl7.ExtractHL7Attributes.onTrigger(ExtractHL7Attributes.java:199)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
>
[jira] [Commented] (MINIFICPP-610) Generate type descriptions
[ https://issues.apache.org/jira/browse/MINIFICPP-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622525#comment-16622525 ] ASF GitHub Bot commented on MINIFICPP-610: -- Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/399 works as advertised. verified build and functionality after rebasing on master > Generate type descriptions > -- > > Key: MINIFICPP-610 > URL: https://issues.apache.org/jira/browse/MINIFICPP-610 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #399: MINIFICPP-610: Add description to types
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/399 works as advertised. verified build and functionality after rebasing on master ---
[jira] [Created] (MINIFICPP-615) Provide Windows OpenSSL Support
Mr TheSegfault created MINIFICPP-615: Summary: Provide Windows OpenSSL Support Key: MINIFICPP-615 URL: https://issues.apache.org/jira/browse/MINIFICPP-615 Project: NiFi MiNiFi C++ Issue Type: Bug Environment: Microsoft Windows Reporter: Mr TheSegfault Assignee: Mr TheSegfault Fix For: 0.6.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-595) Provide support for running on Windows
[ https://issues.apache.org/jira/browse/MINIFICPP-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-595: - Description: provides basic support > Provide support for running on Windows > --- > > Key: MINIFICPP-595 > URL: https://issues.apache.org/jira/browse/MINIFICPP-595 > Project: NiFi MiNiFi C++ > Issue Type: Epic >Reporter: Aldrin Piri >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > provides basic support -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-603) Fill gaps in C2 responses for Windows
[ https://issues.apache.org/jira/browse/MINIFICPP-603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-603: - Fix Version/s: 0.6.0 > Fill gaps in C2 responses for Windows > - > > Key: MINIFICPP-603 > URL: https://issues.apache.org/jira/browse/MINIFICPP-603 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > C2 responses aren't functionally complete in windows. This ticket is meant to > fill those gaps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-595) Provide support for running on Windows
[ https://issues.apache.org/jira/browse/MINIFICPP-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault resolved MINIFICPP-595. -- Resolution: Fixed > Provide support for running on Windows > --- > > Key: MINIFICPP-595 > URL: https://issues.apache.org/jira/browse/MINIFICPP-595 > Project: NiFi MiNiFi C++ > Issue Type: Epic >Reporter: Aldrin Piri >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-603) Fill gaps in C2 responses for Windows
[ https://issues.apache.org/jira/browse/MINIFICPP-603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-603: - Status: Patch Available (was: Open) > Fill gaps in C2 responses for Windows > - > > Key: MINIFICPP-603 > URL: https://issues.apache.org/jira/browse/MINIFICPP-603 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > C2 responses aren't functionally complete in windows. This ticket is meant to > fill those gaps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFIREG-200) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFIREG-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Doran updated NIFIREG-200: Fix Version/s: 0.3.0 > Upgrade version of Jetty > > > Key: NIFIREG-200 > URL: https://issues.apache.org/jira/browse/NIFIREG-200 > Project: NiFi Registry > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Andy LoPresto >Assignee: Kevin Doran >Priority: Blocker > Labels: jetty > Fix For: 0.3.0 > > > Spoke with Kevin off-list; he will make this change. > Please upgrade Jetty to version 9.4.11.x. > See [NIFI-5479|https://issues.apache.org/jira/browse/NIFI-5479]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFIREG-200) Upgrade version of Jetty
Andy LoPresto created NIFIREG-200: - Summary: Upgrade version of Jetty Key: NIFIREG-200 URL: https://issues.apache.org/jira/browse/NIFIREG-200 Project: NiFi Registry Issue Type: Improvement Affects Versions: 0.2.0 Reporter: Andy LoPresto Assignee: Kevin Doran Spoke with Kevin off-list; he will make this change. Please upgrade Jetty to version 9.4.11.x. See [NIFI-5479|https://issues.apache.org/jira/browse/NIFI-5479]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5616) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto resolved NIFI-5616. - Resolution: Invalid > Upgrade version of Jetty > > > Key: NIFI-5616 > URL: https://issues.apache.org/jira/browse/NIFI-5616 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Blocker > Fix For: 1.8.0 > > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5616) Upgrade version of Jetty
Andy LoPresto created NIFI-5616: --- Summary: Upgrade version of Jetty Key: NIFI-5616 URL: https://issues.apache.org/jira/browse/NIFI-5616 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.7.1 Reporter: Andy LoPresto Assignee: Matt Gilman Fix For: 1.8.0 Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5616) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622497#comment-16622497 ] Andy LoPresto commented on NIFI-5616: - Was trying to clone this to NIFIREG; has to be in the same project. Closing. > Upgrade version of Jetty > > > Key: NIFI-5616 > URL: https://issues.apache.org/jira/browse/NIFI-5616 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Blocker > Fix For: 1.8.0 > > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5560) Sub directory(symbolic link to directory) files are not getting listed in ListSFTP(ListSFTP does not Follow symbolic links)
[
https://issues.apache.org/jira/browse/NIFI-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622479#comment-16622479
]
ASF GitHub Bot commented on NIFI-5560:
--
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3000#discussion_r219269381
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/SFTPTransfer.java
---
@@ -281,6 +289,19 @@ public int select(final LsEntry entry) {
logger.error("Unable to get listing from " +
newFullForwardPath + "; skipping this subdirectory", e);
}
}
+
--- End diff --
submitted changes with 52a3432e3b0ad379d63386e4c4b554db35b1e2a3
> Sub directory(symbolic link to directory) files are not getting listed in
> ListSFTP(ListSFTP does not Follow symbolic links)
> ---
>
> Key: NIFI-5560
> URL: https://issues.apache.org/jira/browse/NIFI-5560
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.1
>Reporter: Hemantha kumara M S
>Assignee: Hemantha kumara M S
>Priority: Major
>
> *Here is the configuration*
>
> *SFTP Server side:*
> -bash-4.2$ mkdir -p /tmp/testData
> -bash-4.2$
> -bash-4.2$ mkdir -p /tmp/toRead
> -bash-4.2$ ln -s /tmp/testData /tmp/toRead/data1
> -bash-4.2$ touch /tmp/testData/1.txt
> -bash-4.2$ touch /tmp/testData/2.txt
> -bash-4.2$ touch /tmp/toRead/t.txt
> -bash-4.2$ mkdir /tmp/toRead/data2
> -bash-4.2$ touch /tmp/toRead/data2/22.txt
> -bash-4.2$ cd /tmp/toRead/
> -bash-4.2$ tree
> .
> ├── data1 -> /tmp/testData
> ├── data2
> │ └── 22.txt
> └── t.txt
> 2 directories, 2 files
> -bash-4.2$ pwd
> /tmp/toRead
> -bash-4.2$ tree
> .
> ├── data1 -> /tmp/testData
> ├── data2
> │ └── 22.txt
> └── t.txt
> 2 directories, 2 files
> -bash-4.2$ touch data
> data1/ data2/
> -bash-4.2$ touch data2/22.txt
> -bash-4.2$ touch t.txt
> -bash-4.2$ tree /tmp/testData
> /tmp/testData
> ├── 1.txt
> └── 2.txt
> 0 directories, 2 files
>
> *Nifi:*
> Configured ListSFTP +Remote Path+ to +/tmp/toRead/+ and +Search Recursively+
> to +true+
>
> *+Expected result:+*
> Should list 4 files(1.txt, 2.txt, t.txt, data2/22.txt)
> *+Actual result:+*
> listed only two files(t.txt, data2/22.txt)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5560) Sub directory(symbolic link to directory) files are not getting listed in ListSFTP(ListSFTP does not Follow symbolic links)
[
https://issues.apache.org/jira/browse/NIFI-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622475#comment-16622475
]
ASF GitHub Bot commented on NIFI-5560:
--
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3000#discussion_r219269231
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/SFTPTransfer.java
---
@@ -236,6 +238,12 @@ public int select(final LsEntry entry) {
return LsEntrySelector.CONTINUE;
}
+// if is a link and we're supposed to follow symlink
+if (symlink && entry.getAttrs().isLink()) {
+links.add(entry);
+return LsEntrySelector.CONTINUE;
+}
+
--- End diff --
Submitted changes with commit 52a3432e3b0ad379d63386e4c4b554db35b1e2a3
> Sub directory(symbolic link to directory) files are not getting listed in
> ListSFTP(ListSFTP does not Follow symbolic links)
> ---
>
> Key: NIFI-5560
> URL: https://issues.apache.org/jira/browse/NIFI-5560
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.1
>Reporter: Hemantha kumara M S
>Assignee: Hemantha kumara M S
>Priority: Major
>
> *Here is the configuration*
>
> *SFTP Server side:*
> -bash-4.2$ mkdir -p /tmp/testData
> -bash-4.2$
> -bash-4.2$ mkdir -p /tmp/toRead
> -bash-4.2$ ln -s /tmp/testData /tmp/toRead/data1
> -bash-4.2$ touch /tmp/testData/1.txt
> -bash-4.2$ touch /tmp/testData/2.txt
> -bash-4.2$ touch /tmp/toRead/t.txt
> -bash-4.2$ mkdir /tmp/toRead/data2
> -bash-4.2$ touch /tmp/toRead/data2/22.txt
> -bash-4.2$ cd /tmp/toRead/
> -bash-4.2$ tree
> .
> ├── data1 -> /tmp/testData
> ├── data2
> │ └── 22.txt
> └── t.txt
> 2 directories, 2 files
> -bash-4.2$ pwd
> /tmp/toRead
> -bash-4.2$ tree
> .
> ├── data1 -> /tmp/testData
> ├── data2
> │ └── 22.txt
> └── t.txt
> 2 directories, 2 files
> -bash-4.2$ touch data
> data1/ data2/
> -bash-4.2$ touch data2/22.txt
> -bash-4.2$ touch t.txt
> -bash-4.2$ tree /tmp/testData
> /tmp/testData
> ├── 1.txt
> └── 2.txt
> 0 directories, 2 files
>
> *Nifi:*
> Configured ListSFTP +Remote Path+ to +/tmp/toRead/+ and +Search Recursively+
> to +true+
>
> *+Expected result:+*
> Should list 4 files(1.txt, 2.txt, t.txt, data2/22.txt)
> *+Actual result:+*
> listed only two files(t.txt, data2/22.txt)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3000: NIFI-5560 Added Follow SYMLINK support for ListFTP ...
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3000#discussion_r219269381
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/SFTPTransfer.java
---
@@ -281,6 +289,19 @@ public int select(final LsEntry entry) {
logger.error("Unable to get listing from " +
newFullForwardPath + "; skipping this subdirectory", e);
}
}
+
--- End diff --
submitted changes with 52a3432e3b0ad379d63386e4c4b554db35b1e2a3
---
[jira] [Commented] (NIFIREG-197) Flow and Bucket is not retrieved when cloned from from Git to NiFi Registry
[ https://issues.apache.org/jira/browse/NIFIREG-197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622474#comment-16622474 ] Kevin Doran commented on NIFIREG-197: - [~Varshinee] The steps you describe are actually not a supported means of replicating flows from one NiFi Registry instance to another. The reason is that there is also metadata for buckets and flows that is stored outside of the GitFlowPersistenceProvider. Currently, to migrate flows it is recommended to use the NiFi CLI, which has export, import, and transfer commands. If you would like another way of migrating flows, or initializing a NiFi Registry instance from a git repository, I suggest we close this ticket and open a new ticket as a feature request. > Flow and Bucket is not retrieved when cloned from from Git to NiFi Registry > --- > > Key: NIFIREG-197 > URL: https://issues.apache.org/jira/browse/NIFIREG-197 > Project: NiFi Registry > Issue Type: Bug >Affects Versions: 0.2.0 >Reporter: Varshinee >Priority: Major > > What I did - > # I have two NiFi Registry for Dev and Prod. I have pushed the flow from Dev > NiFi Registry to Git by using GitFlowPersistenceProvider. > # Tried to retrieve the flow and bucket from Git for the Prod machine by git > clone. > # When checking Prod NiFi Registry the flows are not available. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3000: NIFI-5560 Added Follow SYMLINK support for ListFTP ...
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3000#discussion_r219269231
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/SFTPTransfer.java
---
@@ -236,6 +238,12 @@ public int select(final LsEntry entry) {
return LsEntrySelector.CONTINUE;
}
+// if is a link and we're supposed to follow symlink
+if (symlink && entry.getAttrs().isLink()) {
+links.add(entry);
+return LsEntrySelector.CONTINUE;
+}
+
--- End diff --
Submitted changes with commit 52a3432e3b0ad379d63386e4c4b554db35b1e2a3
---
[jira] [Commented] (NIFI-5560) Sub directory(symbolic link to directory) files are not getting listed in ListSFTP(ListSFTP does not Follow symbolic links)
[
https://issues.apache.org/jira/browse/NIFI-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622472#comment-16622472
]
ASF GitHub Bot commented on NIFI-5560:
--
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3000#discussion_r219269070
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FTPTransfer.java
---
@@ -263,6 +265,16 @@ public String getHomeDirectory(final FlowFile
flowFile) throws IOException {
}
}
+// if is a link and we're supposed to follow symlink
+if (symlink && file.isSymbolicLink()) {
+try {
+listing.addAll(getListing(newFullForwardPath, depth +
1, maxResults - count));
+} catch (final IOException e) {
+logger.error("Unable to get listing from " +
newFullForwardPath + "; skipping this symlink", e);
+}
+}
+
+
--- End diff --
Submitted changes with latest commit
52a3432e3b0ad379d63386e4c4b554db35b1e2a3
> Sub directory(symbolic link to directory) files are not getting listed in
> ListSFTP(ListSFTP does not Follow symbolic links)
> ---
>
> Key: NIFI-5560
> URL: https://issues.apache.org/jira/browse/NIFI-5560
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.1
>Reporter: Hemantha kumara M S
>Assignee: Hemantha kumara M S
>Priority: Major
>
> *Here is the configuration*
>
> *SFTP Server side:*
> -bash-4.2$ mkdir -p /tmp/testData
> -bash-4.2$
> -bash-4.2$ mkdir -p /tmp/toRead
> -bash-4.2$ ln -s /tmp/testData /tmp/toRead/data1
> -bash-4.2$ touch /tmp/testData/1.txt
> -bash-4.2$ touch /tmp/testData/2.txt
> -bash-4.2$ touch /tmp/toRead/t.txt
> -bash-4.2$ mkdir /tmp/toRead/data2
> -bash-4.2$ touch /tmp/toRead/data2/22.txt
> -bash-4.2$ cd /tmp/toRead/
> -bash-4.2$ tree
> .
> ├── data1 -> /tmp/testData
> ├── data2
> │ └── 22.txt
> └── t.txt
> 2 directories, 2 files
> -bash-4.2$ pwd
> /tmp/toRead
> -bash-4.2$ tree
> .
> ├── data1 -> /tmp/testData
> ├── data2
> │ └── 22.txt
> └── t.txt
> 2 directories, 2 files
> -bash-4.2$ touch data
> data1/ data2/
> -bash-4.2$ touch data2/22.txt
> -bash-4.2$ touch t.txt
> -bash-4.2$ tree /tmp/testData
> /tmp/testData
> ├── 1.txt
> └── 2.txt
> 0 directories, 2 files
>
> *Nifi:*
> Configured ListSFTP +Remote Path+ to +/tmp/toRead/+ and +Search Recursively+
> to +true+
>
> *+Expected result:+*
> Should list 4 files(1.txt, 2.txt, t.txt, data2/22.txt)
> *+Actual result:+*
> listed only two files(t.txt, data2/22.txt)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3000: NIFI-5560 Added Follow SYMLINK support for ListFTP ...
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3000#discussion_r219269070
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FTPTransfer.java
---
@@ -263,6 +265,16 @@ public String getHomeDirectory(final FlowFile
flowFile) throws IOException {
}
}
+// if is a link and we're supposed to follow symlink
+if (symlink && file.isSymbolicLink()) {
+try {
+listing.addAll(getListing(newFullForwardPath, depth +
1, maxResults - count));
+} catch (final IOException e) {
+logger.error("Unable to get listing from " +
newFullForwardPath + "; skipping this symlink", e);
+}
+}
+
+
--- End diff --
Submitted changes with latest commit
52a3432e3b0ad379d63386e4c4b554db35b1e2a3
---
[jira] [Commented] (NIFIREG-186) Create Authorizer implementation that uses Apache Ranger
[ https://issues.apache.org/jira/browse/NIFIREG-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622423#comment-16622423 ] ASF GitHub Bot commented on NIFIREG-186: Github user asfgit closed the pull request at: https://github.com/apache/nifi-registry/pull/131 > Create Authorizer implementation that uses Apache Ranger > > > Key: NIFIREG-186 > URL: https://issues.apache.org/jira/browse/NIFIREG-186 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > Fix For: 0.3.0 > > > In addition to the standard file-based Authorizer, we should provide an > Authorizer implementation that uses Apache Ranger, so that users implement > centralized authorization against both NiFi and NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFIREG-186) Create Authorizer implementation that uses Apache Ranger
[ https://issues.apache.org/jira/browse/NIFIREG-186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Doran resolved NIFIREG-186. - Resolution: Fixed > Create Authorizer implementation that uses Apache Ranger > > > Key: NIFIREG-186 > URL: https://issues.apache.org/jira/browse/NIFIREG-186 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > Fix For: 0.3.0 > > > In addition to the standard file-based Authorizer, we should provide an > Authorizer implementation that uses Apache Ranger, so that users implement > centralized authorization against both NiFi and NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry pull request #131: NIFIREG-186: Adding Ranger authorizer
Github user asfgit closed the pull request at: https://github.com/apache/nifi-registry/pull/131 ---
[jira] [Commented] (NIFIREG-186) Create Authorizer implementation that uses Apache Ranger
[ https://issues.apache.org/jira/browse/NIFIREG-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622405#comment-16622405 ] ASF GitHub Bot commented on NIFIREG-186: Github user kevdoran commented on a diff in the pull request: https://github.com/apache/nifi-registry/pull/131#discussion_r219256607 --- Diff: nifi-registry-extensions/nifi-registry-ranger/nifi-registry-ranger-assembly/README.md --- @@ -0,0 +1,131 @@ + +# NiFi Registry Ranger extension + +This extension provides `org.apache.nifi.registry.ranger.RangerAuthorizer` class for NiFi Registry to authorize user requests by access policies defined at [Apache Ranger](https://ranger.apache.org/). + +## Prerequisites + +* Apache Ranger 1.2.0 or later is needed. --- End diff -- Ok, looks like this is good for Ranger 1.2.0 and 2.0.0. I'll merge it as-is then, thanks! > Create Authorizer implementation that uses Apache Ranger > > > Key: NIFIREG-186 > URL: https://issues.apache.org/jira/browse/NIFIREG-186 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > Fix For: 0.3.0 > > > In addition to the standard file-based Authorizer, we should provide an > Authorizer implementation that uses Apache Ranger, so that users implement > centralized authorization against both NiFi and NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry pull request #131: NIFIREG-186: Adding Ranger authorizer
Github user kevdoran commented on a diff in the pull request: https://github.com/apache/nifi-registry/pull/131#discussion_r219256607 --- Diff: nifi-registry-extensions/nifi-registry-ranger/nifi-registry-ranger-assembly/README.md --- @@ -0,0 +1,131 @@ + +# NiFi Registry Ranger extension + +This extension provides `org.apache.nifi.registry.ranger.RangerAuthorizer` class for NiFi Registry to authorize user requests by access policies defined at [Apache Ranger](https://ranger.apache.org/). + +## Prerequisites + +* Apache Ranger 1.2.0 or later is needed. --- End diff -- Ok, looks like this is good for Ranger 1.2.0 and 2.0.0. I'll merge it as-is then, thanks! ---
[jira] [Commented] (NIFI-5560) Sub directory(symbolic link to directory) files are not getting listed in ListSFTP(ListSFTP does not Follow symbolic links)
[
https://issues.apache.org/jira/browse/NIFI-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622392#comment-16622392
]
ASF GitHub Bot commented on NIFI-5560:
--
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3000#discussion_r219254765
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FTPTransfer.java
---
@@ -263,6 +265,16 @@ public String getHomeDirectory(final FlowFile
flowFile) throws IOException {
}
}
+// if is a link and we're supposed to follow symlink
+if (symlink && file.isSymbolicLink()) {
+try {
+listing.addAll(getListing(newFullForwardPath, depth +
1, maxResults - count));
+} catch (final IOException e) {
+logger.error("Unable to get listing from " +
newFullForwardPath + "; skipping this symlink", e);
+}
+}
+
+
--- End diff --
yes, I will resubmit with proposed changes.
> Sub directory(symbolic link to directory) files are not getting listed in
> ListSFTP(ListSFTP does not Follow symbolic links)
> ---
>
> Key: NIFI-5560
> URL: https://issues.apache.org/jira/browse/NIFI-5560
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.1
>Reporter: Hemantha kumara M S
>Assignee: Hemantha kumara M S
>Priority: Major
>
> *Here is the configuration*
>
> *SFTP Server side:*
> -bash-4.2$ mkdir -p /tmp/testData
> -bash-4.2$
> -bash-4.2$ mkdir -p /tmp/toRead
> -bash-4.2$ ln -s /tmp/testData /tmp/toRead/data1
> -bash-4.2$ touch /tmp/testData/1.txt
> -bash-4.2$ touch /tmp/testData/2.txt
> -bash-4.2$ touch /tmp/toRead/t.txt
> -bash-4.2$ mkdir /tmp/toRead/data2
> -bash-4.2$ touch /tmp/toRead/data2/22.txt
> -bash-4.2$ cd /tmp/toRead/
> -bash-4.2$ tree
> .
> ├── data1 -> /tmp/testData
> ├── data2
> │ └── 22.txt
> └── t.txt
> 2 directories, 2 files
> -bash-4.2$ pwd
> /tmp/toRead
> -bash-4.2$ tree
> .
> ├── data1 -> /tmp/testData
> ├── data2
> │ └── 22.txt
> └── t.txt
> 2 directories, 2 files
> -bash-4.2$ touch data
> data1/ data2/
> -bash-4.2$ touch data2/22.txt
> -bash-4.2$ touch t.txt
> -bash-4.2$ tree /tmp/testData
> /tmp/testData
> ├── 1.txt
> └── 2.txt
> 0 directories, 2 files
>
> *Nifi:*
> Configured ListSFTP +Remote Path+ to +/tmp/toRead/+ and +Search Recursively+
> to +true+
>
> *+Expected result:+*
> Should list 4 files(1.txt, 2.txt, t.txt, data2/22.txt)
> *+Actual result:+*
> listed only two files(t.txt, data2/22.txt)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3000: NIFI-5560 Added Follow SYMLINK support for ListFTP ...
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3000#discussion_r219254765
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FTPTransfer.java
---
@@ -263,6 +265,16 @@ public String getHomeDirectory(final FlowFile
flowFile) throws IOException {
}
}
+// if is a link and we're supposed to follow symlink
+if (symlink && file.isSymbolicLink()) {
+try {
+listing.addAll(getListing(newFullForwardPath, depth +
1, maxResults - count));
+} catch (final IOException e) {
+logger.error("Unable to get listing from " +
newFullForwardPath + "; skipping this symlink", e);
+}
+}
+
+
--- End diff --
yes, I will resubmit with proposed changes.
---
[jira] [Commented] (NIFI-5560) Sub directory(symbolic link to directory) files are not getting listed in ListSFTP(ListSFTP does not Follow symbolic links)
[
https://issues.apache.org/jira/browse/NIFI-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622384#comment-16622384
]
ASF GitHub Bot commented on NIFI-5560:
--
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3000#discussion_r219253879
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FileTransfer.java
---
@@ -141,6 +141,13 @@ default String getAbsolutePath(FlowFile flowFile,
String remotePath) throws IOEx
.defaultValue("false")
.allowableValues("true", "false")
.build();
+public static final PropertyDescriptor FOLLOW_SYMLINK = new
PropertyDescriptor.Builder()
--- End diff --
as this property is common for both ftp and sftp, will it be good if I
mention only ftp-follow-symlink ? as earlier properties like recursive, file
regex and other properties names kept generic hence I had put only Follow
symlink.
> Sub directory(symbolic link to directory) files are not getting listed in
> ListSFTP(ListSFTP does not Follow symbolic links)
> ---
>
> Key: NIFI-5560
> URL: https://issues.apache.org/jira/browse/NIFI-5560
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.1
>Reporter: Hemantha kumara M S
>Assignee: Hemantha kumara M S
>Priority: Major
>
> *Here is the configuration*
>
> *SFTP Server side:*
> -bash-4.2$ mkdir -p /tmp/testData
> -bash-4.2$
> -bash-4.2$ mkdir -p /tmp/toRead
> -bash-4.2$ ln -s /tmp/testData /tmp/toRead/data1
> -bash-4.2$ touch /tmp/testData/1.txt
> -bash-4.2$ touch /tmp/testData/2.txt
> -bash-4.2$ touch /tmp/toRead/t.txt
> -bash-4.2$ mkdir /tmp/toRead/data2
> -bash-4.2$ touch /tmp/toRead/data2/22.txt
> -bash-4.2$ cd /tmp/toRead/
> -bash-4.2$ tree
> .
> ├── data1 -> /tmp/testData
> ├── data2
> │ └── 22.txt
> └── t.txt
> 2 directories, 2 files
> -bash-4.2$ pwd
> /tmp/toRead
> -bash-4.2$ tree
> .
> ├── data1 -> /tmp/testData
> ├── data2
> │ └── 22.txt
> └── t.txt
> 2 directories, 2 files
> -bash-4.2$ touch data
> data1/ data2/
> -bash-4.2$ touch data2/22.txt
> -bash-4.2$ touch t.txt
> -bash-4.2$ tree /tmp/testData
> /tmp/testData
> ├── 1.txt
> └── 2.txt
> 0 directories, 2 files
>
> *Nifi:*
> Configured ListSFTP +Remote Path+ to +/tmp/toRead/+ and +Search Recursively+
> to +true+
>
> *+Expected result:+*
> Should list 4 files(1.txt, 2.txt, t.txt, data2/22.txt)
> *+Actual result:+*
> listed only two files(t.txt, data2/22.txt)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3000: NIFI-5560 Added Follow SYMLINK support for ListFTP ...
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3000#discussion_r219253879
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FileTransfer.java
---
@@ -141,6 +141,13 @@ default String getAbsolutePath(FlowFile flowFile,
String remotePath) throws IOEx
.defaultValue("false")
.allowableValues("true", "false")
.build();
+public static final PropertyDescriptor FOLLOW_SYMLINK = new
PropertyDescriptor.Builder()
--- End diff --
as this property is common for both ftp and sftp, will it be good if I
mention only ftp-follow-symlink ? as earlier properties like recursive, file
regex and other properties names kept generic hence I had put only Follow
symlink.
---
