[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478043#comment-16478043
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2693


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.7.0
>
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477890#comment-16477890
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2693
  
Thanks @markap14! This has been merged to master.


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
> Fix For: 1.7.0
>
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476316#comment-16476316
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2693
  
@mcgilman I pushed a new commit that i think should address the feedback. 
Thanks!


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475930#comment-16475930
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2693
  
@markap14 There appears to be a regression in the updates to the cluster 
replication logic. The behavior I'm seeing is that a request to remove a 
property is ignored. I'm guessing the changes to (de)serialization causes this 
part of the request to be lost.


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474366#comment-16474366
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2693#discussion_r187999206
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/StandardAuthorizableLookup.java
 ---
@@ -493,10 +493,6 @@ public Authorizable getAuthorizableFromResource(String 
resource) {
 }
 }
 
-if (resourceType == null) {
--- End diff --

I see. It looks like this is unreachable and resourceType will never be 
null. However, it appears to be a minor bug. I'll propose a minor edit or file 
a separate follow up JIRA for this.


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472549#comment-16472549
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2693#discussion_r187673609
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/authorization/StandardAuthorizableLookup.java
 ---
@@ -493,10 +493,6 @@ public Authorizable getAuthorizableFromResource(String 
resource) {
 }
 }
 
-if (resourceType == null) {
--- End diff --

Why is this being removed?


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472550#comment-16472550
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2693#discussion_r187686929
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestListenHTTP.java
 ---
@@ -168,6 +169,8 @@ public void 
testSecurePOSTRequestsReturnCodeReceivedWithEL() throws Exception {
 }
 
 @Test
+// TODO / NOCOMMIT: Don't check in with this ignored... it's now 
failing because the service is valid. DOn't know why it was invalid before
--- End diff --

Was this meant to be committed :)


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469086#comment-16469086
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2693
  
Will review...


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469051#comment-16469051
 ] 

ASF GitHub Bot commented on NIFI-950:
-

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2693

NIFI-950: Make component validation asynchronous

This PR addresses NIFI-950 as well as a handful of other related JIRAs. I 
used a single PR because a lot of the solutions to the issues built upon one 
another and because the intent of this PR is basically to make clustering more 
stable and to make the UI feel less sluggish when clustered.


Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-950

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2693.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2693


commit b155a7474e98eb597b2417fef276a6bcddf18d78
Author: Mark Payne 
Date:   2018-04-11T19:36:54Z

NIFI-950: Make component validation asynchronous
NIFI-950: Still seeing some slow response times when instantiating a large 
template in cluster mode so making some minor tweaks based on the results of 
CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired 
intermediate data model that can be serialized, separate from serializing. This 
allows us to hold the FlowController's Read Lock only while creating the data 
model, not while actually serializing the data. Configured Jersey Client in 
ThreadPoolRequestReplicator not to look for features using the Service Loader 
for every request. Updated Template object to hold a DOM Node that represents 
the template contents instead of having to serialize the DTO, then parse the 
serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead 
of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it 
clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so 
that it will clean up any resource related to component lifecycle.

commit be01d8e1d2fc985a852ee849bdb4c39639642ae1
Author: Mark Payne 
Date:   2018-04-24T19:52:36Z

NIFI-950: Avoid gathering the Status objects for entire flow when we don't 
need them; removed unnecessary code

commit 1751d276813fa52fa98f9656c712931c130e66f3
Author: Mark Payne 
Date:   2018-05-03T20:07:10Z

NIFI-950: Bug fixes

commit d6e29be47b9befa218df4ecb0af33d7b15df6be2
Author: Mark Payne 
Date:   2018-05-08T17:35:44Z

NIFI-950: Bug fix; added validation status to ProcessorDTO, 
ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause 
time to be set in the customValidate method for testing functionality

commit 352bc4c9eb880f29cb2aefcb29539923f00ad89b
Author: Mark Payne 
Date:   2018-05-08T19:59:43Z

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467889#comment-16467889
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2689
  
@mcgilman I am closing this PR because it needs to be rebased and I ran 
into an issue with a unit test during the rebase. Will open a new PR once that 
has been addressed.


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467890#comment-16467890
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user markap14 closed the pull request at:

https://github.com/apache/nifi/pull/2689


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467774#comment-16467774
 ] 

ASF GitHub Bot commented on NIFI-950:
-

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2689
  
Will review...


> Perform component validation asynchronously
> ---
>
> Key: NIFI-950
> URL: https://issues.apache.org/jira/browse/NIFI-950
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
> Attachments: self_reference_flow_fix.xml
>
>
> I created a flow that is a self referencing http loop. The flow was working 
> fine but I wanted to save the template for later testing. I downloaded the 
> the flow as a template. Then I tried testing a thread.sleep in the beginning 
> of onConfigured, createSSLContext, and validate methods of 
> StandardSSLContextService. I did a mvn clean install in the 
> nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service
>  directory. Then a mvn clean install in the nifi-assembly directory. After I 
> imported the template the UI became very slow when clicking to different 
> windows of the UI such as configuring a processor and the controller services 
> window.
> I then stashed my changes and rebuilt the files. Once again I imported my 
> template, and attempting to configure a processor or accessing the controller 
> services window became very slow.
> The flow xml is attached. 
> ---
> The description and attachment showed an issue where long running validation 
> caused the UI to become unresponsive. This validation should be done 
> asynchronously so that the UI always remains responsive. Initial thoughts...
> - new state to indicate that validation is in progress
> - a mechanism for refreshing validation results
> - time out for waiting for validation to complete? or need to always be 
> validating all components in case their validity is based on something 
> environmental (like a configuration file that is modified outside of the 
> application)?
> - provide better support for components that are running and become invalid
> -- related to this we need to provide guidance regarding the difference 
> between become invalid and when we should use features like bulletins and 
> yielding to rely runtime issues



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-950) Perform component validation asynchronously

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467771#comment-16467771
 ] 

ASF GitHub Bot commented on NIFI-950:
-

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2689

NIFI-950: Perform component validation asynchronously, other 
cluster-related performance enhancements

This PR addresses NIFI-950 and a handful of other JIRA's. They were all 
done in a single PR because the modifications tend to build upon one another, 
and also because this PR really addresses the issue of making nifi clustering 
more responsive (in terms of UI) and more stable.


Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-950

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2689.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2689


commit 05f77401395b4274f2b09bf43b812af325e81690
Author: Mark Payne 
Date:   2018-04-11T19:36:54Z

NIFI-950: Make component validation asynchronous
NIFI-950: Still seeing some slow response times when instantiating a large 
template in cluster mode so making some minor tweaks based on the results of 
CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired 
intermediate data model that can be serialized, separate from serializing. This 
allows us to hold the FlowController's Read Lock only while creating the data 
model, not while actually serializing the data. Configured Jersey Client in 
ThreadPoolRequestReplicator not to look for features using the Service Loader 
for every request. Updated Template object to hold a DOM Node that represents 
the template contents instead of having to serialize the DTO, then parse the 
serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead 
of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it 
clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so 
that it will clean up any resource related to component lifecycle.

commit 3ef86dc0c4e2c4855972cd0c200b7b9982e0c040
Author: Mark Payne 
Date:   2018-04-24T19:52:36Z

NIFI-950: Avoid gathering the Status objects for entire flow when we don't 
need them; removed unnecessary code

commit a7c7fcb7c3fc72f8fc0a32e6ebdc4795bc12a8b4
Author: Mark Payne 
Date:   2018-05-03T20:07:10Z

NIFI-950: Bug fixes

commit 434aa1dd85c53a7cc58497aad42246692a19a739
Author: Mark Payne 
Date:   2018-05-08T17:35:44Z

NIFI-950: Bug fix; added validation status to ProcessorDTO, 
ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause 
time to be set in the customValidate method for testing functionality




> Perform component validation asynchronously
>