[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128254#comment-16128254
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133351212
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -219,6 +234,25 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 }
 
 final ComponentLog logger = getLogger();
+final XMLReader xmlReader;
+
+try {
+xmlReader = XMLReaderFactory.createXMLReader();
+} catch (SAXException e) {
+logger.error("Error while constructing XMLReader {}", new 
Object[]{e});
+session.transfer(flowFiles, REL_FAILURE);
--- End diff --

EvaluateXQuery throws ProcessException. EvaluateXPath should do the same.


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133351212
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -219,6 +234,25 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 }
 
 final ComponentLog logger = getLogger();
+final XMLReader xmlReader;
+
+try {
+xmlReader = XMLReaderFactory.createXMLReader();
+} catch (SAXException e) {
+logger.error("Error while constructing XMLReader {}", new 
Object[]{e});
+session.transfer(flowFiles, REL_FAILURE);
--- End diff --

EvaluateXQuery throws ProcessException. EvaluateXPath should do the same.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133350881
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestEvaluateXPath.java
 ---
@@ -16,23 +16,23 @@
  */
 package org.apache.nifi.processors.standard;
 
-import static org.junit.Assert.assertTrue;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
 
+import javax.xml.xpath.XPathFactoryConfigurationException;
 import java.io.IOException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 
-import javax.xml.xpath.XPathFactoryConfigurationException;
-
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-
-import org.junit.Test;
+import static org.junit.Assert.assertTrue;
--- End diff --

These import statement change can be cleaner, too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128251#comment-16128251
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133350949
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestEvaluateXQuery.java
 ---
@@ -65,7 +65,6 @@ public void testSetTransformerProperties() throws 
Exception {
 }
 }
 
-@Ignore("this test is failing")
--- End diff --

Thanks for fixing this :)


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128253#comment-16128253
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133350881
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestEvaluateXPath.java
 ---
@@ -16,23 +16,23 @@
  */
 package org.apache.nifi.processors.standard;
 
-import static org.junit.Assert.assertTrue;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
 
+import javax.xml.xpath.XPathFactoryConfigurationException;
 import java.io.IOException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 
-import javax.xml.xpath.XPathFactoryConfigurationException;
-
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-
-import org.junit.Test;
+import static org.junit.Assert.assertTrue;
--- End diff --

These import statement change can be cleaner, too.


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128252#comment-16128252
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133350909
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestEvaluateXQuery.java
 ---
@@ -31,19 +35,15 @@
 import java.util.Map.Entry;
 import java.util.Properties;
 
-import javax.xml.transform.OutputKeys;
-import javax.xml.xpath.XPathFactoryConfigurationException;
-
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.Ignore;
-
-import org.junit.Test;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
--- End diff --

These import statement change can be cleaner, too.


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133350909
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestEvaluateXQuery.java
 ---
@@ -31,19 +35,15 @@
 import java.util.Map.Entry;
 import java.util.Properties;
 
-import javax.xml.transform.OutputKeys;
-import javax.xml.xpath.XPathFactoryConfigurationException;
-
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.Ignore;
-
-import org.junit.Test;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
--- End diff --

These import statement change can be cleaner, too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133350949
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestEvaluateXQuery.java
 ---
@@ -65,7 +65,6 @@ public void testSetTransformerProperties() throws 
Exception {
 }
 }
 
-@Ignore("this test is failing")
--- End diff --

Thanks for fixing this :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4258) CSVUtils uses same AllowableValue for 'Informix Unload' and 'Informix Unload Escape Disabled'

[Wesley L Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wesley L Lawrence updated NIFI-4258:

Attachment: NIFI-4258.patch

> CSVUtils uses same AllowableValue for 'Informix Unload' and 'Informix Unload 
> Escape Disabled'
> -
>
> Key: NIFI-4258
> URL: https://issues.apache.org/jira/browse/NIFI-4258
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Attachments: NIFI-4258.patch
>
>
> Related to NIFI-4242, if you can't use 'Informix Unload Escape Disabled' as a 
> pre-defined CSV format, because 'Informix Unload' has the same allowable 
> value. 
> The WebUI for CSVRedaer/CSVRecordSetWriter seems to always display 'Informix 
> Unload', and when choosing from the drop down, says 'Informix Unload Escape 
> Delimited'. 
> Given that within CSVUtils, 'Informix Unload' is checked against first, I 
> suspect that's the one that gets chosen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4258) CSVUtils uses same AllowableValue for 'Informix Unload' and 'Informix Unload Escape Disabled'

[Wesley L Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wesley L Lawrence updated NIFI-4258:

Status: Patch Available  (was: Open)

> CSVUtils uses same AllowableValue for 'Informix Unload' and 'Informix Unload 
> Escape Disabled'
> -
>
> Key: NIFI-4258
> URL: https://issues.apache.org/jira/browse/NIFI-4258
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Attachments: NIFI-4258.patch
>
>
> Related to NIFI-4242, if you can't use 'Informix Unload Escape Disabled' as a 
> pre-defined CSV format, because 'Informix Unload' has the same allowable 
> value. 
> The WebUI for CSVRedaer/CSVRecordSetWriter seems to always display 'Informix 
> Unload', and when choosing from the drop down, says 'Informix Unload Escape 
> Delimited'. 
> Given that within CSVUtils, 'Informix Unload' is checked against first, I 
> suspect that's the one that gets chosen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4258) CSVUtils uses same AllowableValue for 'Informix Unload' and 'Informix Unload Escape Disabled'

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127860#comment-16127860
 ] 

ASF GitHub Bot commented on NIFI-4258:
--

GitHub user Wesley-Lawrence opened a pull request:

https://github.com/apache/nifi/pull/2090

NIFI-4258 Corrected a bug with the 'Informix Unload Escape Disabled' CSV 
format.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [✓] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [✓] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [✓] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [✓] Is your initial contribution a single, squashed commit?

### For code changes:
- [✓] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [N/A] Have you written or updated unit tests to verify your changes?
- [N/A] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [N/A] If applicable, have you updated the LICENSE file, including the 
main LICENSE file under nifi-assembly?
- [N/A] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [N/A] If adding new Properties, have you added .displayName in addition 
to .name (programmatic access) for each of the new properties?

### For documentation related changes:
- [N/A] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Wesley-Lawrence/nifi NIFI-4258

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2090.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2090


commit 7171f5f7ea9482228f04b519577e3355047f3fc6
Author: Wesley-Lawrence 
Date:   2017-08-15T20:53:27Z

NIFI-4258 Corrected a bug with the 'Informix Unload Escape Disabled' CSV 
format.




> CSVUtils uses same AllowableValue for 'Informix Unload' and 'Informix Unload 
> Escape Disabled'
> -
>
> Key: NIFI-4258
> URL: https://issues.apache.org/jira/browse/NIFI-4258
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Wesley L Lawrence
>Priority: Minor
>
> Related to NIFI-4242, if you can't use 'Informix Unload Escape Disabled' as a 
> pre-defined CSV format, because 'Informix Unload' has the same allowable 
> value. 
> The WebUI for CSVRedaer/CSVRecordSetWriter seems to always display 'Informix 
> Unload', and when choosing from the drop down, says 'Informix Unload Escape 
> Delimited'. 
> Given that within CSVUtils, 'Informix Unload' is checked against first, I 
> suspect that's the one that gets chosen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2090: NIFI-4258 Corrected a bug with the 'Informix Unload...

[Wesley-Lawrence]

GitHub user Wesley-Lawrence opened a pull request:

https://github.com/apache/nifi/pull/2090

NIFI-4258 Corrected a bug with the 'Informix Unload Escape Disabled' CSV 
format.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [✓] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [✓] Does your PR title start with NIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [✓] Has your PR been rebased against the latest commit within the 
target branch (typically master)?

- [✓] Is your initial contribution a single, squashed commit?

### For code changes:
- [✓] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [N/A] Have you written or updated unit tests to verify your changes?
- [N/A] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [N/A] If applicable, have you updated the LICENSE file, including the 
main LICENSE file under nifi-assembly?
- [N/A] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [N/A] If adding new Properties, have you added .displayName in addition 
to .name (programmatic access) for each of the new properties?

### For documentation related changes:
- [N/A] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Wesley-Lawrence/nifi NIFI-4258

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2090.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2090


commit 7171f5f7ea9482228f04b519577e3355047f3fc6
Author: Wesley-Lawrence 
Date:   2017-08-15T20:53:27Z

NIFI-4258 Corrected a bug with the 'Informix Unload Escape Disabled' CSV 
format.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4242) CSVReader shouldn't require that an escape character be defined

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127794#comment-16127794
 ] 

ASF GitHub Bot commented on NIFI-4242:
--

GitHub user Wesley-Lawrence opened a pull request:

https://github.com/apache/nifi/pull/2088

NIFI-4242 Allow quote and escape chars for CSV to be 'undefined'.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [✓] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [✓] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [✓] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [✓] Is your initial contribution a single, squashed commit?

### For code changes:
- [✓] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [✓] Have you written or updated unit tests to verify your changes?
- [N/A] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [N/A] If applicable, have you updated the LICENSE file, including the 
main LICENSE file under nifi-assembly?
- [N/A] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [N/A] If adding new Properties, have you added .displayName in addition 
to .name (programmatic access) for each of the new properties?

### For documentation related changes:
- [N/A] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Wesley-Lawrence/nifi NIFI-4242

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2088.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2088


commit fbe8ad616297f7a6bf0b17039c9448fb2ac1514c
Author: Wesley-Lawrence 
Date:   2017-08-15T19:28:09Z

NIFI-4242 Allow quote and escape chars for CSV to be 'undefined'.




> CSVReader shouldn't require that an escape character be defined
> ---
>
> Key: NIFI-4242
> URL: https://issues.apache.org/jira/browse/NIFI-4242
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Attachments: NIFI-4242.patch, NIFI-4242.patch
>
>
> There are situations where, when parsing a CSV file, one doesn't want to 
> define an escape character. For example, when using quote character ", the 
> following is valid CSV;
> {code}
> a,"""b",c
> {code}
> The second column should be interpreted as "b. But when Apache Commons CSV is 
> told that there's an escape character, the above row is invalid 
> (interestingly, if it was """b""", it would be valid as "b"). 
> There are known formats that Apache Commons CSV provides, that doesn't define 
> escape characters either.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2089: Nifi-ldap-iaa support for PasswordComparisonAuthent...

[felixalbani]

GitHub user felixalbani opened a pull request:

https://github.com/apache/nifi/pull/2089

Nifi-ldap-iaa support for PasswordComparisonAuthenticator

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixalbani/nifi-1 NIFI-4293

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2089.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2089


commit 3c8749d5d72d742f65c1890f6f60cd6f58dbe906
Author: Felix Albani 
Date:   2017-08-15T18:47:23Z

Nifi-ldap-iaa support for PasswordComparisonAuthenticator




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2088: NIFI-4242 Allow quote and escape chars for CSV to b...

[Wesley-Lawrence]

GitHub user Wesley-Lawrence opened a pull request:

https://github.com/apache/nifi/pull/2088

NIFI-4242 Allow quote and escape chars for CSV to be 'undefined'.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [✓] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [✓] Does your PR title start with NIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [✓] Has your PR been rebased against the latest commit within the 
target branch (typically master)?

- [✓] Is your initial contribution a single, squashed commit?

### For code changes:
- [✓] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [✓] Have you written or updated unit tests to verify your changes?
- [N/A] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [N/A] If applicable, have you updated the LICENSE file, including the 
main LICENSE file under nifi-assembly?
- [N/A] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [N/A] If adding new Properties, have you added .displayName in addition 
to .name (programmatic access) for each of the new properties?

### For documentation related changes:
- [N/A] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Wesley-Lawrence/nifi NIFI-4242

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2088.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2088


commit fbe8ad616297f7a6bf0b17039c9448fb2ac1514c
Author: Wesley-Lawrence 
Date:   2017-08-15T19:28:09Z

NIFI-4242 Allow quote and escape chars for CSV to be 'undefined'.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4242) CSVReader shouldn't require that an escape character be defined

[Wesley L Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wesley L Lawrence updated NIFI-4242:

Attachment: NIFI-4242.patch

> CSVReader shouldn't require that an escape character be defined
> ---
>
> Key: NIFI-4242
> URL: https://issues.apache.org/jira/browse/NIFI-4242
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Attachments: NIFI-4242.patch, NIFI-4242.patch
>
>
> There are situations where, when parsing a CSV file, one doesn't want to 
> define an escape character. For example, when using quote character ", the 
> following is valid CSV;
> {code}
> a,"""b",c
> {code}
> The second column should be interpreted as "b. But when Apache Commons CSV is 
> told that there's an escape character, the above row is invalid 
> (interestingly, if it was """b""", it would be valid as "b"). 
> There are known formats that Apache Commons CSV provides, that doesn't define 
> escape characters either.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4301) ExecuteScript Processor executing Python Script fails at os.getpid()

[Will Lieu (JIRA)]

Will Lieu created NIFI-4301:
---

 Summary: ExecuteScript Processor executing Python Script fails at 
os.getpid()
 Key: NIFI-4301
 URL: https://issues.apache.org/jira/browse/NIFI-4301
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.3.0
Reporter: Will Lieu
Priority: Blocker
 Fix For: 1.4.0, 1.3.0


Currently NiFi Version 1.3.0 uses Jython-Shaded-2.7.0 which contains a bug of 
the os.getpid() method not being implemented. Is there any way you guys can rev 
this jar to use 2.7.1? 

See: [Jython Issue 2405|http://bugs.jython.org/issue2405.)]





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-4281) ExecuteScript with ruby throws IndexOutOfBoundsException on all scripts

[Jeremy Lautman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremy Lautman resolved NIFI-4281.
--
Resolution: Not A Bug

> ExecuteScript with ruby throws IndexOutOfBoundsException on all scripts
> ---
>
> Key: NIFI-4281
> URL: https://issues.apache.org/jira/browse/NIFI-4281
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Jeremy Lautman
>Priority: Critical
> Attachments: Test_ExecuteScript_Ruby_JW1.xml
>
>
> In running ExecuteScript with the following ruby script, ExecuteScript throws 
> IndexOutOfBoundsExceptions.
> {quote}session.transfer(session.get(), REL_SUCCESS){quote}
> The same script with python works appropriately.
> It also does the same when using the session::get(int) variant, and when 
> running larger scripts.
> NiFi version 1.3.0
> Java openjdk version 1.8.0_121



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-minifi-cpp issue #127: MINIFI-375: Remove forward slash from urls

[apiri]

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/127
  
reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #127: MINIFI-375: Remove forward slash from urls

[apiri]

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/127
  
Was able to recreate the issue and verify that this resolved it.  This 
makes sense as to why it was not working given the resource paths for the NiFi 
endpoints.  Will merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4242) CSVReader shouldn't require that an escape character be defined

[Wesley L Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wesley L Lawrence updated NIFI-4242:

Status: Patch Available  (was: Open)

> CSVReader shouldn't require that an escape character be defined
> ---
>
> Key: NIFI-4242
> URL: https://issues.apache.org/jira/browse/NIFI-4242
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Attachments: NIFI-4242.patch
>
>
> There are situations where, when parsing a CSV file, one doesn't want to 
> define an escape character. For example, when using quote character ", the 
> following is valid CSV;
> {code}
> a,"""b",c
> {code}
> The second column should be interpreted as "b. But when Apache Commons CSV is 
> told that there's an escape character, the above row is invalid 
> (interestingly, if it was """b""", it would be valid as "b"). 
> There are known formats that Apache Commons CSV provides, that doesn't define 
> escape characters either.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4242) CSVReader shouldn't require that an escape character be defined

[Wesley L Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wesley L Lawrence updated NIFI-4242:

Attachment: NIFI-4242.patch

> CSVReader shouldn't require that an escape character be defined
> ---
>
> Key: NIFI-4242
> URL: https://issues.apache.org/jira/browse/NIFI-4242
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Attachments: NIFI-4242.patch
>
>
> There are situations where, when parsing a CSV file, one doesn't want to 
> define an escape character. For example, when using quote character ", the 
> following is valid CSV;
> {code}
> a,"""b",c
> {code}
> The second column should be interpreted as "b. But when Apache Commons CSV is 
> told that there's an escape character, the above row is invalid 
> (interestingly, if it was """b""", it would be valid as "b"). 
> There are known formats that Apache Commons CSV provides, that doesn't define 
> escape characters either.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-minifi-cpp pull request #127: MINIFI-375: Remove forward slash from url...

[phrocker]

GitHub user phrocker opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/127

MINIFI-375: Remove forward slash from urls

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFI-375

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/127.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #127


commit 01cc006be5926e680b7089d6a9c8f016008dc773
Author: Marc Parisi 
Date:   2017-08-15T19:05:41Z

MINIFI-375: Remove forward slash from urls




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3869) add http/2 support to HTTP listening processors

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127720#comment-16127720
 ] 

Michael Hogue commented on NIFI-3869:
-

Work here: https://github.com/m-hogue/nifi/tree/NIFI-3869

> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Tony Kurc
>Assignee: Michael Hogue
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3869) add http/2 support to HTTP listening processors

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127703#comment-16127703
 ] 

Michael Hogue commented on NIFI-3869:
-

[~trk...@gmail.com] : At a quick glance, i see two HTTP listening processors: 
{{ListenHTTP}} and {{HandleHTTPRequest}}. Do you know of more?

> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Tony Kurc
>Assignee: Michael Hogue
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3869) add http/2 support to HTTP listening processors

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3869:
---

Assignee: Michael Hogue

> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Tony Kurc
>Assignee: Michael Hogue
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4281) ExecuteScript with ruby throws IndexOutOfBoundsException on all scripts

[Jeremy Lautman (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127549#comment-16127549
 ] 

Jeremy Lautman commented on NIFI-4281:
--

Thanks for the template! I was able to get the template working on my system, 
which means I must have been doing something else wrong. This ticket can 
probably be closed out.

> ExecuteScript with ruby throws IndexOutOfBoundsException on all scripts
> ---
>
> Key: NIFI-4281
> URL: https://issues.apache.org/jira/browse/NIFI-4281
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Jeremy Lautman
>Priority: Critical
> Attachments: Test_ExecuteScript_Ruby_JW1.xml
>
>
> In running ExecuteScript with the following ruby script, ExecuteScript throws 
> IndexOutOfBoundsExceptions.
> {quote}session.transfer(session.get(), REL_SUCCESS){quote}
> The same script with python works appropriately.
> It also does the same when using the session::get(int) variant, and when 
> running larger scripts.
> NiFi version 1.3.0
> Java openjdk version 1.8.0_121



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4295) When selecting a controller service for a processor, services that belong to the wrong scope are shown

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4295:
-
Status: Patch Available  (was: In Progress)

> When selecting a controller service for a processor, services that belong to 
> the wrong scope are shown
> --
>
> Key: NIFI-4295
> URL: https://issues.apache.org/jira/browse/NIFI-4295
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> I have a Process Group, and in that Process Group I have defined a Controller 
> Service. If I then move up to the root group and add a Processor, that takes 
> a Controller Service of that type, I can now choose the Controller Service 
> that is defined in a child group. When I do so, my processor becomes invalid 
> and the reason given for the processor to be invalid is a NullPointer:
> {code}
> 2017-08-14 12:38:24,016 WARN [NiFi Web Server-76] 
> o.a.n.controller.StandardProcessorNode Failed during validation
> java.lang.NullPointerException: null
>   at 
> org.apache.nifi.processor.StandardValidationContext.isValidationRequired(StandardValidationContext.java:143)
>   at 
> org.apache.nifi.components.PropertyDescriptor.validate(PropertyDescriptor.java:150)
>   at 
> org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:103)
>   at 
> org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:329)
>   at 
> org.apache.nifi.controller.StandardProcessorNode.isValid(StandardProcessorNode.java:968)
>   at 
> org.apache.nifi.controller.FlowController.getProcessorStatus(FlowController.java:2964)
>   at 
> org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2559)
>   at 
> org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2518)
>   at 
> org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2485)
>   at 
> org.apache.nifi.web.controller.ControllerFacade.getProcessGroupStatus(ControllerFacade.java:599)
>   at 
> org.apache.nifi.web.controller.ControllerFacade$$FastClassBySpringCGLIB$$5a42ba54.invoke()
>   at 
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
>   at 
> org.apache.nifi.web.controller.ControllerFacade$$EnhancerBySpringCGLIB$$344e3ba2.getProcessGroupStatus()
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade.getProcessGroupFlow(StandardNiFiServiceFacade.java:3054)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
>   at 
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>   at 
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithReadLock(NiFiServiceFacadeLock.java:137)
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.getLock(NiFiServiceFacadeLock.java:108)
>   at sun.reflect.GeneratedMethodAccessor168.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>   at 
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> 

[jira] [Updated] (NIFI-4201) Add Processors for interacting with Kafka 0.11.x

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4201:
-
Fix Version/s: 1.4.0
   Status: Patch Available  (was: Open)

> Add Processors for interacting with Kafka 0.11.x
> 
>
> Key: NIFI-4201
> URL: https://issues.apache.org/jira/browse/NIFI-4201
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4295) When selecting a controller service for a processor, services that belong to the wrong scope are shown

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127510#comment-16127510
 ] 

ASF GitHub Bot commented on NIFI-4295:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2087

NIFI-4295: When determining which controller services to return for a…

… component, ensure that we don't show services that belong to 'child 
groups'

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4295

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2087.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2087


commit fd9d71dbf412746de2cc36ddb91c59b05f8b4134
Author: Mark Payne 
Date:   2017-08-15T16:48:30Z

NIFI-4295: When determining which controller services to return for a 
component, ensure that we don't show services that belong to 'child groups'




> When selecting a controller service for a processor, services that belong to 
> the wrong scope are shown
> --
>
> Key: NIFI-4295
> URL: https://issues.apache.org/jira/browse/NIFI-4295
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> I have a Process Group, and in that Process Group I have defined a Controller 
> Service. If I then move up to the root group and add a Processor, that takes 
> a Controller Service of that type, I can now choose the Controller Service 
> that is defined in a child group. When I do so, my processor becomes invalid 
> and the reason given for the processor to be invalid is a NullPointer:
> {code}
> 2017-08-14 12:38:24,016 WARN [NiFi Web Server-76] 
> o.a.n.controller.StandardProcessorNode Failed during validation
> java.lang.NullPointerException: null
>   at 
> org.apache.nifi.processor.StandardValidationContext.isValidationRequired(StandardValidationContext.java:143)
>   at 
> org.apache.nifi.components.PropertyDescriptor.validate(PropertyDescriptor.java:150)
>   at 
> org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:103)
>   at 
> org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:329)
>   at 
> org.apache.nifi.controller.StandardProcessorNode.isValid(StandardProcessorNode.java:968)
>   at 
> org.apache.nifi.controller.FlowController.getProcessorStatus(FlowController.java:2964)
>   at 
> org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2559)
>   at 
> org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2518)
>   at 
> 

[GitHub] nifi pull request #2087: NIFI-4295: When determining which controller servic...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2087

NIFI-4295: When determining which controller services to return for a…

… component, ensure that we don't show services that belong to 'child 
groups'

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4295

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2087.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2087


commit fd9d71dbf412746de2cc36ddb91c59b05f8b4134
Author: Mark Payne 
Date:   2017-08-15T16:48:30Z

NIFI-4295: When determining which controller services to return for a 
component, ensure that we don't show services that belong to 'child groups'




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (NIFI-4198) *ElasticsearchHttp processors do not expose Proxy settings

[Arun Manivannan (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127497#comment-16127497
 ] 

Arun Manivannan edited comment on NIFI-4198 at 8/15/17 4:38 PM:


[~trixpan] Managed to create an integration testing environment for easier 
testing with squid on docker, as suggested by [~joewitt]. Patch is on the way. 
https://github.com/arunma/elastic-behind-a-proxy


was (Author: arunodhaya80):
[~trixpan] Managed to create an integration testing environment for easier 
testing. Patch is on the way. 
https://github.com/arunma/elastic-behind-a-proxy

> *ElasticsearchHttp processors do not expose Proxy settings
> --
>
> Key: NIFI-4198
> URL: https://issues.apache.org/jira/browse/NIFI-4198
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre F de Miranda
>Assignee: Arun Manivannan
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4198) *ElasticsearchHttp processors do not expose Proxy settings

[Arun Manivannan (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127497#comment-16127497
 ] 

Arun Manivannan commented on NIFI-4198:
---

[~trixpan] Managed to create an integration testing environment for easier 
testing. Patch is on the way. 
https://github.com/arunma/elastic-behind-a-proxy

> *ElasticsearchHttp processors do not expose Proxy settings
> --
>
> Key: NIFI-4198
> URL: https://issues.apache.org/jira/browse/NIFI-4198
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre F de Miranda
>Assignee: Arun Manivannan
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4295) When selecting a controller service for a processor, services that belong to the wrong scope are shown

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4295:
-
Description: 
I have a Process Group, and in that Process Group I have defined a Controller 
Service. If I then move up to the root group and add a Processor, that takes a 
Controller Service of that type, I can now choose the Controller Service that 
is defined in a child group. When I do so, my processor becomes invalid and the 
reason given for the processor to be invalid is a NullPointer:

{code}
2017-08-14 12:38:24,016 WARN [NiFi Web Server-76] 
o.a.n.controller.StandardProcessorNode Failed during validation
java.lang.NullPointerException: null
at 
org.apache.nifi.processor.StandardValidationContext.isValidationRequired(StandardValidationContext.java:143)
at 
org.apache.nifi.components.PropertyDescriptor.validate(PropertyDescriptor.java:150)
at 
org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:103)
at 
org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:329)
at 
org.apache.nifi.controller.StandardProcessorNode.isValid(StandardProcessorNode.java:968)
at 
org.apache.nifi.controller.FlowController.getProcessorStatus(FlowController.java:2964)
at 
org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2559)
at 
org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2518)
at 
org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2485)
at 
org.apache.nifi.web.controller.ControllerFacade.getProcessGroupStatus(ControllerFacade.java:599)
at 
org.apache.nifi.web.controller.ControllerFacade$$FastClassBySpringCGLIB$$5a42ba54.invoke()
at 
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at 
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at 
org.apache.nifi.web.controller.ControllerFacade$$EnhancerBySpringCGLIB$$344e3ba2.getProcessGroupStatus()
at 
org.apache.nifi.web.StandardNiFiServiceFacade.getProcessGroupFlow(StandardNiFiServiceFacade.java:3054)
at 
org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
at 
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at 
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at 
org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithReadLock(NiFiServiceFacadeLock.java:137)
at 
org.apache.nifi.web.NiFiServiceFacadeLock.getLock(NiFiServiceFacadeLock.java:108)
at sun.reflect.GeneratedMethodAccessor168.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at 
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at 
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at 
org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$c5bfd05e.getProcessGroupFlow()
at org.apache.nifi.web.api.FlowResource.getFlow(FlowResource.java:349)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 

[jira] [Updated] (NIFI-4295) When selecting a controller service for a processor, services that belong to the wrong scope are shown

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4295:
-
Priority: Major  (was: Blocker)

> When selecting a controller service for a processor, services that belong to 
> the wrong scope are shown
> --
>
> Key: NIFI-4295
> URL: https://issues.apache.org/jira/browse/NIFI-4295
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> I have a Process Group, and in that Process Group I have defined a Controller 
> Service. If I then move up to the root group and add a Processor, that takes 
> a Controller Service of that type, I can now choose the Controller Service 
> that is defined in a child group. When I do so, my processor becomes invalid 
> and the reason given for the processor to be invalid is a NullPointer:
> {code}
> 2017-08-14 12:38:24,016 WARN [NiFi Web Server-76] 
> o.a.n.controller.StandardProcessorNode Failed during validation
> java.lang.NullPointerException: null
>   at 
> org.apache.nifi.processor.StandardValidationContext.isValidationRequired(StandardValidationContext.java:143)
>   at 
> org.apache.nifi.components.PropertyDescriptor.validate(PropertyDescriptor.java:150)
>   at 
> org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:103)
>   at 
> org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:329)
>   at 
> org.apache.nifi.controller.StandardProcessorNode.isValid(StandardProcessorNode.java:968)
>   at 
> org.apache.nifi.controller.FlowController.getProcessorStatus(FlowController.java:2964)
>   at 
> org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2559)
>   at 
> org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2518)
>   at 
> org.apache.nifi.controller.FlowController.getGroupStatus(FlowController.java:2485)
>   at 
> org.apache.nifi.web.controller.ControllerFacade.getProcessGroupStatus(ControllerFacade.java:599)
>   at 
> org.apache.nifi.web.controller.ControllerFacade$$FastClassBySpringCGLIB$$5a42ba54.invoke()
>   at 
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
>   at 
> org.apache.nifi.web.controller.ControllerFacade$$EnhancerBySpringCGLIB$$344e3ba2.getProcessGroupStatus()
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade.getProcessGroupFlow(StandardNiFiServiceFacade.java:3054)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
>   at 
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>   at 
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithReadLock(NiFiServiceFacadeLock.java:137)
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.getLock(NiFiServiceFacadeLock.java:108)
>   at sun.reflect.GeneratedMethodAccessor168.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>   at 
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> 

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127473#comment-16127473
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133234914
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -309,18 +349,18 @@ public void process(final InputStream rawIn) throws 
IOException {
 }
 
 if (returnType == NODESET) {
-List nodeList = (List) result;
-if (nodeList.isEmpty()) {
+DOMNodeList nodeList = (DOMNodeList) result;
+if (nodeList.getLength()==0) {
--- End diff --

Fixed by removing the line.


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[arunma]

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133234914
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -309,18 +349,18 @@ public void process(final InputStream rawIn) throws 
IOException {
 }
 
 if (returnType == NODESET) {
-List nodeList = (List) result;
-if (nodeList.isEmpty()) {
+DOMNodeList nodeList = (DOMNodeList) result;
+if (nodeList.getLength()==0) {
--- End diff --

Fixed by removing the line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[arunma]

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133234692
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -277,8 +316,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 @Override
 public void process(final InputStream rawIn) throws 
IOException {
 try (final InputStream in = new 
BufferedInputStream(rawIn)) {
-final List rootList = (List) 
slashExpression.evaluate(new InputSource(in), NODESET);
-sourceRef.set(rootList.get(0));
+final Document document=docBuilder.parse(in);
--- End diff --

Fixed by removing the line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[arunma]

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133234584
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -219,6 +239,25 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 }
 
 final ComponentLog logger = getLogger();
+final DocumentBuilderFactory 
docBuilderFactory=docBuilderFactoryRef.get();
+final DocumentBuilder docBuilder;
+try {
+docBuilder = docBuilderFactory.newDocumentBuilder();
+
+if (!context.getProperty(VALIDATE_DTD).asBoolean()) {
+docBuilder.setEntityResolver(new EntityResolver() {
+@Override
+public InputSource resolveEntity(String publicId, 
String systemId) throws SAXException, IOException {
+return new InputSource(new StringReader(""));
+}
+});
+}
+} catch (ParserConfigurationException e) {
+logger.error("Configuration error while constructing 
DocumentBuilder {}", new Object[]{e});
+session.transfer(flowFiles, REL_FAILURE);
+return;
--- End diff --

@ijokarumawak Brilliant. That makes a lot of sense and I understand the 
spirit of it. Also, with the new change, there is a place where I am 
instantiating the XMLReader and that throws a SAX Exception.  I feel that there 
is no way to recover from it and have transferred the flow to FAILURE.  Let me 
know if that ought to be changed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127464#comment-16127464
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133234173
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -219,6 +239,25 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 }
 
 final ComponentLog logger = getLogger();
+final DocumentBuilderFactory 
docBuilderFactory=docBuilderFactoryRef.get();
--- End diff --

Removed the line :-)


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127463#comment-16127463
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133234113
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -199,8 +216,11 @@ protected void init(final 
ProcessorInitializationContext context) {
 }
 
 @OnScheduled
-public void initializeXPathFactory() throws 
XPathFactoryConfigurationException {
+public void initializeFactory() throws 
XPathFactoryConfigurationException {
 
factoryRef.set(XPathFactory.newInstance(NamespaceConstant.OBJECT_MODEL_SAXON));
+final DocumentBuilderFactory 
docBuilderFactory=DocumentBuilderFactory.newInstance();
--- End diff --

With the new changes, these lines were removed but I'll watch out for 
spaces around '='


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[arunma]

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133234173
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -219,6 +239,25 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 }
 
 final ComponentLog logger = getLogger();
+final DocumentBuilderFactory 
docBuilderFactory=docBuilderFactoryRef.get();
--- End diff --

Removed the line :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[arunma]

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133234113
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -199,8 +216,11 @@ protected void init(final 
ProcessorInitializationContext context) {
 }
 
 @OnScheduled
-public void initializeXPathFactory() throws 
XPathFactoryConfigurationException {
+public void initializeFactory() throws 
XPathFactoryConfigurationException {
 
factoryRef.set(XPathFactory.newInstance(NamespaceConstant.OBJECT_MODEL_SAXON));
+final DocumentBuilderFactory 
docBuilderFactory=DocumentBuilderFactory.newInstance();
--- End diff --

With the new changes, these lines were removed but I'll watch out for 
spaces around '='


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4208) Node failed to join cluster due to NullPointerException

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-4208:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Node failed to join cluster due to NullPointerException
> ---
>
> Key: NIFI-4208
> URL: https://issues.apache.org/jira/browse/NIFI-4208
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.4.0
>
>
> A clustered node ran out of disk space. Upon restart, I came across the 
> following error:
> 2017-07-20 09:03:00,988 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:945)
>   at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515)
>   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
>   at org.apache.nifi.NiFi.(NiFi.java:160)
>   at org.apache.nifi.NiFi.main(NiFi.java:267)
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:43)
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:26)
>   at 
> org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1132)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>   at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:381)
>   at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:713)
>   at 
> org.apache.nifi.controller.StandardFlowService.initializeController(StandardFlowService.java:955)
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:927)
>   ... 4 common frames omitted



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4208) Node failed to join cluster due to NullPointerException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127448#comment-16127448
 ] 

ASF GitHub Bot commented on NIFI-4208:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2086


> Node failed to join cluster due to NullPointerException
> ---
>
> Key: NIFI-4208
> URL: https://issues.apache.org/jira/browse/NIFI-4208
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.4.0
>
>
> A clustered node ran out of disk space. Upon restart, I came across the 
> following error:
> 2017-07-20 09:03:00,988 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:945)
>   at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515)
>   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
>   at org.apache.nifi.NiFi.(NiFi.java:160)
>   at org.apache.nifi.NiFi.main(NiFi.java:267)
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:43)
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:26)
>   at 
> org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1132)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>   at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:381)
>   at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:713)
>   at 
> org.apache.nifi.controller.StandardFlowService.initializeController(StandardFlowService.java:955)
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:927)
>   ... 4 common frames omitted



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2086: NIFI-4208: Fixed bug in SchemaRepositoryRecordSerde...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2086


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4208) Node failed to join cluster due to NullPointerException

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127447#comment-16127447
 ] 

ASF subversion and git services commented on NIFI-4208:
---

Commit abca9d1464fe0d6a5deefb28260465a310a97bd4 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=abca9d1 ]

NIFI-4208: Fixed bug in SchemaRepositoryRecordSerde that would return null from 
deserializeEdit if there was no data; the interface documents that null cannot 
be returned from this method, as it is called only when data is expected to 
exist. As a result, if there is no data, we should throw EOFException instead, 
and the write-ahead log will handle that appropriately.

This closes #2086.

Signed-off-by: Andy LoPresto 


> Node failed to join cluster due to NullPointerException
> ---
>
> Key: NIFI-4208
> URL: https://issues.apache.org/jira/browse/NIFI-4208
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.4.0
>
>
> A clustered node ran out of disk space. Upon restart, I came across the 
> following error:
> 2017-07-20 09:03:00,988 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:945)
>   at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515)
>   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
>   at org.apache.nifi.NiFi.(NiFi.java:160)
>   at org.apache.nifi.NiFi.main(NiFi.java:267)
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:43)
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:26)
>   at 
> org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1132)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>   at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:381)
>   at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:713)
>   at 
> org.apache.nifi.controller.StandardFlowService.initializeController(StandardFlowService.java:955)
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:927)
>   ... 4 common frames omitted



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4208) Node failed to join cluster due to NullPointerException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127446#comment-16127446
 ] 

ASF GitHub Bot commented on NIFI-4208:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2086
  
I verified the new unit test passed successfully. I then started the app 
normally, ran a heavy flow, and debugged at the `checkpoint()` method to ensure 
I captured a valid `snapshot` file. I modified this by truncating the file 
after a transaction ID, yielding an incomplete record. I stopped the 
application, moved the modified `snapshot` into place, and restarted the app. 
On loading, NiFi stopped with the following expected error in the 
`nifi-app.log`:

```
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
at 
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88)
at 
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876)
at 
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
at 
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839)
at 
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
at 
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480)
at 
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442)
at 
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799)
at 
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at 
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:452)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:419)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:707)
at org.apache.nifi.NiFi.(NiFi.java:160)
at org.apache.nifi.NiFi.main(NiFi.java:268)
Caused by: java.io.EOFException: Failed to read field 'Repository Record 
Update'
at 
org.apache.nifi.repository.schema.SchemaRecordReader.readField(SchemaRecordReader.java:115)
at 
org.apache.nifi.repository.schema.SchemaRecordReader.readRecord(SchemaRecordReader.java:72)
at 
org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:124)
at 
org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:46)
at 
org.wali.MinimalLockingWriteAheadLog.recoverFromSnapshot(MinimalLockingWriteAheadLog.java:380)
at 
org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:300)
at 
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:381)
at 
org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:723)
at 
org.apache.nifi.controller.StandardFlowService.initializeController(StandardFlowService.java:956)
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:535)
at 
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
... 28 common frames omitted
Suppressed: 

[GitHub] nifi issue #2086: NIFI-4208: Fixed bug in SchemaRepositoryRecordSerde that w...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2086
  
I verified the new unit test passed successfully. I then started the app 
normally, ran a heavy flow, and debugged at the `checkpoint()` method to ensure 
I captured a valid `snapshot` file. I modified this by truncating the file 
after a transaction ID, yielding an incomplete record. I stopped the 
application, moved the modified `snapshot` into place, and restarted the app. 
On loading, NiFi stopped with the following expected error in the 
`nifi-app.log`:

```
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
at 
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88)
at 
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876)
at 
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
at 
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839)
at 
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
at 
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480)
at 
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442)
at 
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799)
at 
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at 
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:452)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:419)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:707)
at org.apache.nifi.NiFi.(NiFi.java:160)
at org.apache.nifi.NiFi.main(NiFi.java:268)
Caused by: java.io.EOFException: Failed to read field 'Repository Record 
Update'
at 
org.apache.nifi.repository.schema.SchemaRecordReader.readField(SchemaRecordReader.java:115)
at 
org.apache.nifi.repository.schema.SchemaRecordReader.readRecord(SchemaRecordReader.java:72)
at 
org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:124)
at 
org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:46)
at 
org.wali.MinimalLockingWriteAheadLog.recoverFromSnapshot(MinimalLockingWriteAheadLog.java:380)
at 
org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:300)
at 
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:381)
at 
org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:723)
at 
org.apache.nifi.controller.StandardFlowService.initializeController(StandardFlowService.java:956)
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:535)
at 
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
... 28 common frames omitted
Suppressed: java.io.EOFException: Failed to read field 'Create or 
Update'
at 
org.apache.nifi.repository.schema.SchemaRecordReader.readField(SchemaRecordReader.java:115)
at 

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127369#comment-16127369
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133217990
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -125,6 +132,14 @@
 .defaultValue(RETURN_TYPE_AUTO)
 .build();
 
+public static final PropertyDescriptor VALIDATE_DTD = new 
PropertyDescriptor.Builder()
+.name("Validate DTD")
+.description("Specifies whether or not the XML content should 
be validated against the DTD.")
+.required(false)
--- End diff --

Ah.  I didn't foresee the "No value".  Thanks. Fixed it. 


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[arunma]

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133217990
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -125,6 +132,14 @@
 .defaultValue(RETURN_TYPE_AUTO)
 .build();
 
+public static final PropertyDescriptor VALIDATE_DTD = new 
PropertyDescriptor.Builder()
+.name("Validate DTD")
+.description("Specifies whether or not the XML content should 
be validated against the DTD.")
+.required(false)
--- End diff --

Ah.  I didn't foresee the "No value".  Thanks. Fixed it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127366#comment-16127366
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133217585
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -75,10 +45,47 @@
 import org.apache.nifi.processor.io.OutputStreamCallback;
 import org.apache.nifi.stream.io.BufferedInputStream;
 import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.w3c.dom.Document;
+import org.xml.sax.EntityResolver;
 import org.xml.sax.InputSource;
+import org.xml.sax.SAXException;
 
-import net.sf.saxon.lib.NamespaceConstant;
-import net.sf.saxon.xpath.XPathEvaluator;
+import javax.xml.namespace.QName;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.ErrorListener;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Source;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.TransformerFactoryConfigurationError;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+import javax.xml.xpath.XPathExpression;
+import javax.xml.xpath.XPathExpressionException;
+import javax.xml.xpath.XPathFactory;
+import javax.xml.xpath.XPathFactoryConfigurationException;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.StringReader;
+import java.io.UnsupportedEncodingException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static javax.xml.xpath.XPathConstants.NODESET;
+import static javax.xml.xpath.XPathConstants.STRING;
--- End diff --

Thanks a lot for the review @ijokarumawak.  I just figured out a way to do 
it in SAX and the change is much smaller.  Passes all the testcases as well. 
Pushing it along with the other changes in a few minutes.

Yes, reverted the imports back and applied only the changes.  It's the IDE 
:-)


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[arunma]

Github user arunma commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133217585
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -75,10 +45,47 @@
 import org.apache.nifi.processor.io.OutputStreamCallback;
 import org.apache.nifi.stream.io.BufferedInputStream;
 import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.w3c.dom.Document;
+import org.xml.sax.EntityResolver;
 import org.xml.sax.InputSource;
+import org.xml.sax.SAXException;
 
-import net.sf.saxon.lib.NamespaceConstant;
-import net.sf.saxon.xpath.XPathEvaluator;
+import javax.xml.namespace.QName;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.ErrorListener;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Source;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.TransformerFactoryConfigurationError;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+import javax.xml.xpath.XPathExpression;
+import javax.xml.xpath.XPathExpressionException;
+import javax.xml.xpath.XPathFactory;
+import javax.xml.xpath.XPathFactoryConfigurationException;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.StringReader;
+import java.io.UnsupportedEncodingException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static javax.xml.xpath.XPathConstants.NODESET;
+import static javax.xml.xpath.XPathConstants.STRING;
--- End diff --

Thanks a lot for the review @ijokarumawak.  I just figured out a way to do 
it in SAX and the change is much smaller.  Passes all the testcases as well. 
Pushing it along with the other changes in a few minutes.

Yes, reverted the imports back and applied only the changes.  It's the IDE 
:-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4208) Node failed to join cluster due to NullPointerException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127326#comment-16127326
 ] 

ASF GitHub Bot commented on NIFI-4208:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2086
  
Reviewing...


> Node failed to join cluster due to NullPointerException
> ---
>
> Key: NIFI-4208
> URL: https://issues.apache.org/jira/browse/NIFI-4208
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.4.0
>
>
> A clustered node ran out of disk space. Upon restart, I came across the 
> following error:
> 2017-07-20 09:03:00,988 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:945)
>   at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515)
>   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
>   at org.apache.nifi.NiFi.(NiFi.java:160)
>   at org.apache.nifi.NiFi.main(NiFi.java:267)
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:43)
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:26)
>   at 
> org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1132)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>   at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:381)
>   at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:713)
>   at 
> org.apache.nifi.controller.StandardFlowService.initializeController(StandardFlowService.java:955)
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:927)
>   ... 4 common frames omitted



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2086: NIFI-4208: Fixed bug in SchemaRepositoryRecordSerde that w...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2086
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4208) Node failed to join cluster due to NullPointerException

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4208:
-
Status: Patch Available  (was: Open)

> Node failed to join cluster due to NullPointerException
> ---
>
> Key: NIFI-4208
> URL: https://issues.apache.org/jira/browse/NIFI-4208
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.4.0
>
>
> A clustered node ran out of disk space. Upon restart, I came across the 
> following error:
> 2017-07-20 09:03:00,988 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:945)
>   at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515)
>   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
>   at org.apache.nifi.NiFi.(NiFi.java:160)
>   at org.apache.nifi.NiFi.main(NiFi.java:267)
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:43)
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:26)
>   at 
> org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1132)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>   at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:381)
>   at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:713)
>   at 
> org.apache.nifi.controller.StandardFlowService.initializeController(StandardFlowService.java:955)
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:927)
>   ... 4 common frames omitted



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4208) Node failed to join cluster due to NullPointerException

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127321#comment-16127321
 ] 

Mark Payne commented on NIFI-4208:
--

[~joewitt] - sorry, just figured out what was going on here. Looks like we were 
returning a null when the interface documents that null is not allowed and 
instead we should throw EOFException (because we've already found some data for 
the record). Posted a PR to address. The nice thing is that the bug affects 
reading the data, not writing it. As a result, if a user runs into this issue, 
the patch should allow the NiFi instance to startup properly, once applied.

> Node failed to join cluster due to NullPointerException
> ---
>
> Key: NIFI-4208
> URL: https://issues.apache.org/jira/browse/NIFI-4208
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.4.0
>
>
> A clustered node ran out of disk space. Upon restart, I came across the 
> following error:
> 2017-07-20 09:03:00,988 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:945)
>   at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515)
>   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
>   at org.apache.nifi.NiFi.(NiFi.java:160)
>   at org.apache.nifi.NiFi.main(NiFi.java:267)
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:43)
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:26)
>   at 
> org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1132)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
>   at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>   at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:381)
>   at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:713)
>   at 
> org.apache.nifi.controller.StandardFlowService.initializeController(StandardFlowService.java:955)
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:927)
>   ... 4 common frames omitted



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4208) Node failed to join cluster due to NullPointerException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127314#comment-16127314
 ] 

ASF GitHub Bot commented on NIFI-4208:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2086

NIFI-4208: Fixed bug in SchemaRepositoryRecordSerde that would return…

… null from deserializeEdit if there was no data; the interface documents 
that null cannot be returned from this method, as it is called only when data 
is expected to exist. As a result, if there is no data, we should throw 
EOFException instead, and the write-ahead log will handle that appropriately

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4208

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2086.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2086


commit a8cf805381489b35a56062a0cdf52eea6fa4fb76
Author: Mark Payne 
Date:   2017-08-15T14:39:46Z

NIFI-4208: Fixed bug in SchemaRepositoryRecordSerde that would return null 
from deserializeEdit if there was no data; the interface documents that null 
cannot be returned from this method, as it is called only when data is expected 
to exist. As a result, if there is no data, we should throw EOFException 
instead, and the write-ahead log will handle that appropriately




> Node failed to join cluster due to NullPointerException
> ---
>
> Key: NIFI-4208
> URL: https://issues.apache.org/jira/browse/NIFI-4208
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.4.0
>
>
> A clustered node ran out of disk space. Upon restart, I came across the 
> following error:
> 2017-07-20 09:03:00,988 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.lang.NullPointerException
>   at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:945)
>   at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515)
>   at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
>   at org.apache.nifi.NiFi.(NiFi.java:160)
>   at org.apache.nifi.NiFi.main(NiFi.java:267)
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:43)
>   at 
> org.apache.nifi.controller.repository.RepositoryRecordSerde.getRecordIdentifier(RepositoryRecordSerde.java:26)
>   at 
> 

[GitHub] nifi pull request #2086: NIFI-4208: Fixed bug in SchemaRepositoryRecordSerde...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2086

NIFI-4208: Fixed bug in SchemaRepositoryRecordSerde that would return…

… null from deserializeEdit if there was no data; the interface documents 
that null cannot be returned from this method, as it is called only when data 
is expected to exist. As a result, if there is no data, we should throw 
EOFException instead, and the write-ahead log will handle that appropriately

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4208

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2086.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2086


commit a8cf805381489b35a56062a0cdf52eea6fa4fb76
Author: Mark Payne 
Date:   2017-08-15T14:39:46Z

NIFI-4208: Fixed bug in SchemaRepositoryRecordSerde that would return null 
from deserializeEdit if there was no data; the interface documents that null 
cannot be returned from this method, as it is called only when data is expected 
to exist. As a result, if there is no data, we should throw EOFException 
instead, and the write-ahead log will handle that appropriately




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4300) Further review dependency upgrades

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127295#comment-16127295
 ] 

Matt Burgess commented on NIFI-4300:


+1 to Bryan's suggestion

> Further review dependency upgrades
> --
>
> Key: NIFI-4300
> URL: https://issues.apache.org/jira/browse/NIFI-4300
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Extensions
>Affects Versions: 1.3.0
>Reporter: Andy LoPresto
>  Labels: dependencies, security
>
> For further review: 
> * {{org.apache.poi:poi}} in {{nifi-media-nar}} 3.12-beta1 -> 3.15 | Would 
> require upgrading to a new version of tika-core/tika-parses * which have catx 
> json dependencies.
> * {{commons-fileupload:commons-fileupload}} in {{nifi-gcp-nar}} 1.3.1 -> 
> 1.3.2 | Would require upgrading google-cloud but no production * release 
> since}} 0.8.0. Could manually exclude commons-fileupload and directly depend 
> on the newer version.  
> * {{commons-fileupload:commons-fileupload}} in {{nifi-gcp-nar}} 1.3.1 -> 
> 1.3.2 | Would require upgrading google-cloud but no production * release 
> since}} 0.8.0. Could manually exclude commons-fileupload and directly depend 
> on the newer version.  
> * {{commons-collections:commons-collections}} in 
> {{nifi-hbase_1_1_2-client-service}} 3.2.1 -> 3.2.2 | Check with 
> Burgess/Bende. Would * require manual exclusive across multiple dependencies 
> and directly dependency on}} 3.2.2. 
> * {{commons-httpclient:commons-httpclient}} in {{nifi-hdfs-processors}} 3.1 
> -> 4.5.3 | 3.x EOL. Would require hadoop upgrade or manually * exclusive. 
> However, manual exclusive is super risky given the version difference.  
> * {{commons-httpclient:commons-httpclient}} in {{nifi-hdfs-processors}} 3.1 
> -> 4.5.3 | 3.x EOL Would require hadoop upgrade or manually * exclusive. 
> However, manual exclusive is super risky given the version difference.   
> * {{com.fasterxml.jackson.core:jackson-core}} in {{nifi-gcp-nar}} 2.1.3 -> 
> 2.8.6 | Possible manual exclusion, but multiple dependencies * requiring the 
> depender (google-auth-library-oauth2-http).   
> * {{commons-httpclient:commons-httpclient}} in {{nifi-hive-processors}} 3.0.1 
> -> 4.5.3 | 3.x EOL Would require hadoop upgrade or manually * exclusive. 
> However, manual exclusion is super risky given the version difference. 
> * {{commons-httpclient:commons-httpclient}} in {{nifi-hive-processors}} 3.0.1 
> -> 4.5.3 | 3.x EOL Would require hadoop upgrade or manually * exclusive. 
> However, manual exclusion is super risky given the version difference. 
> * {{com.fasterxml.jackson.core:jackson-core}} in 
> {{nifi-elasticsearch-5-processors}} 2.8.1 -> 2.8.6 | Can upgrade to}} 2.8.6 
> of * org.elasticsearch.client:transport}} in {{(and update 
> nifi-expression-language to}} 2.8.6). Confirm with Bende. 
> * {{commons-httpclient:commons-httpclient}} in {{nifi-ranger-nar}} 4.2.5 -> 
> 4.5.3 | Would require manual exclusion through hadoop-common * and 
> hadoop-auth.   
> * {{com.fasterxml.jackson.core:jackson-core}} in {{nifi-spark-receiver}} 
> 2.6.5 -> 2.8.6 | Could update direct dependency on * jackson-databind but 
> would conflict with spark-core_2.10.   
> * {{commons-collections:commons-collections}} in 
> {{nifi-hbase_1_1_2-client-service}} 3.2.1 -> 3.2.2 | Would require manual 
> exclusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3376) Content repository disk usage is not close to reported size in Status Bar

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3376:
---

Assignee: (was: Michael Hogue)

> Content repository disk usage is not close to reported size in Status Bar
> -
>
> Key: NIFI-3376
> URL: https://issues.apache.org/jira/browse/NIFI-3376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Michael Moser
> Attachments: NIFI-3376_Content_Repo_size_demo.xml
>
>
> On NiFi systems that deal with many files whose size is less than 1 MB, we 
> often see that the actual disk usage of the content_repository is much 
> greater than the size of flowfiles that NiFi reports are in its queues.  As 
> an example, NiFi may report "50,000 / 12.5 GB" but the content_repository 
> takes up 240 GB of its file system.  This leads to scenarios where a 500 GB 
> content_repository file system gets 100% full, but "I only had 40 GB of data 
> in my NiFi!"
> When several content claims exist in a single resource claim, and most but 
> not all content claims are terminated, the entire resource claim is still not 
> eligible for deletion or archive.  This could mean that only one 10 KB 
> content claim out of a 1 MB resource claim is counted by NiFi as existing in 
> its queues.
> If a particular flow has a slow egress point where flowfiles could back up 
> and remain on the system longer than expected, this problem is exacerbated.
> A potential solution is to compact resource claim files on disk. A background 
> thread could examine all resource claims, and for those that get "old" and 
> whose active content claim usage drops below a threshold, then rewrite the 
> resource claim file.
> A potential work-around is to allow modification of the FileSystemRepository 
> MAX_APPENDABLE_CLAIM_LENGTH to make it a smaller number.  This would increase 
> the probability that the content claims reference count in a resource claim 
> would reach 0 and the resource claim becomes eligible for deletion/archive.  
> Let users trade-off performance for more accurate accounting of NiFi queue 
> size to content repository size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4210) Add OpenId Connect support for authenticating users

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-4210:
--
Labels: documentation  (was: )

> Add OpenId Connect support for authenticating users
> ---
>
> Key: NIFI-4210
> URL: https://issues.apache.org/jira/browse/NIFI-4210
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>  Labels: documentation
> Fix For: 1.4.0
>
>
> Add support for authenticating users with the OpenId Connection 
> specification. Evaluate whether a new extension point is necessary to allow 
> for a given provider to supply custom code for instance to implement custom 
> token validation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126950#comment-16126950
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133135972
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -125,6 +132,14 @@
 .defaultValue(RETURN_TYPE_AUTO)
 .build();
 
+public static final PropertyDescriptor VALIDATE_DTD = new 
PropertyDescriptor.Builder()
+.name("Validate DTD")
+.description("Specifies whether or not the XML content should 
be validated against the DTD.")
+.required(false)
--- End diff --

A select type property with a default value is usually defined as 
'required(true)'. If not, 'No value' can be selected from the pull down menu, 
but it will be overwritten by the default value. 


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126952#comment-16126952
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133138260
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXQuery.java
 ---
@@ -272,10 +315,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 public void process(final InputStream rawIn) throws 
IOException {
 try (final InputStream in = new 
BufferedInputStream(rawIn)) {
 XQueryEvaluator qe = slashExpression.load();
-qe.setSource(new SAXSource(new InputSource(in)));
-DocumentBuilderFactory dfactory = 
DocumentBuilderFactory.newInstance();
-dfactory.setNamespaceAware(true);
-Document dom = 
dfactory.newDocumentBuilder().newDocument();
+final Document document=docBuilder.parse(in);
+qe.setSource(new DOMSource(document));
+Document dom = docBuilder.newDocument();
--- End diff --

Is there any performance impact by changing from SAX to DOM to evaluate a 
large XML document?


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126956#comment-16126956
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133137297
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -309,18 +349,18 @@ public void process(final InputStream rawIn) throws 
IOException {
 }
 
 if (returnType == NODESET) {
-List nodeList = (List) result;
-if (nodeList.isEmpty()) {
+DOMNodeList nodeList = (DOMNodeList) result;
+if (nodeList.getLength()==0) {
--- End diff --

White space.


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126955#comment-16126955
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133133523
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -75,10 +45,47 @@
 import org.apache.nifi.processor.io.OutputStreamCallback;
 import org.apache.nifi.stream.io.BufferedInputStream;
 import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.w3c.dom.Document;
+import org.xml.sax.EntityResolver;
 import org.xml.sax.InputSource;
+import org.xml.sax.SAXException;
 
-import net.sf.saxon.lib.NamespaceConstant;
-import net.sf.saxon.xpath.XPathEvaluator;
+import javax.xml.namespace.QName;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.ErrorListener;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Source;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.TransformerFactoryConfigurationError;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+import javax.xml.xpath.XPathExpression;
+import javax.xml.xpath.XPathExpressionException;
+import javax.xml.xpath.XPathFactory;
+import javax.xml.xpath.XPathFactoryConfigurationException;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.StringReader;
+import java.io.UnsupportedEncodingException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static javax.xml.xpath.XPathConstants.NODESET;
+import static javax.xml.xpath.XPathConstants.STRING;
--- End diff --

I assume these import statement changes are automatically generated by IDE. 
Is it possible to change only necessary lines of code?


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133135433
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -219,6 +239,25 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 }
 
 final ComponentLog logger = getLogger();
+final DocumentBuilderFactory 
docBuilderFactory=docBuilderFactoryRef.get();
--- End diff --

White space.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126951#comment-16126951
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133137100
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -219,6 +239,25 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 }
 
 final ComponentLog logger = getLogger();
+final DocumentBuilderFactory 
docBuilderFactory=docBuilderFactoryRef.get();
+final DocumentBuilder docBuilder;
+try {
+docBuilder = docBuilderFactory.newDocumentBuilder();
+
+if (!context.getProperty(VALIDATE_DTD).asBoolean()) {
+docBuilder.setEntityResolver(new EntityResolver() {
+@Override
+public InputSource resolveEntity(String publicId, 
String systemId) throws SAXException, IOException {
+return new InputSource(new StringReader(""));
+}
+});
+}
+} catch (ParserConfigurationException e) {
+logger.error("Configuration error while constructing 
DocumentBuilder {}", new Object[]{e});
+session.transfer(flowFiles, REL_FAILURE);
+return;
--- End diff --

Throwing ProcessException would be more appropriate here, because the 
ParserConfigurationException is thrown regardless of incoming FlowFiles. By 
throwing ProcessException, incoming FlowFiles stay in the queue and can be 
processed after the user have fixed the issue.


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126953#comment-16126953
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133135433
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -219,6 +239,25 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 }
 
 final ComponentLog logger = getLogger();
+final DocumentBuilderFactory 
docBuilderFactory=docBuilderFactoryRef.get();
--- End diff --

White space.


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133137100
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -219,6 +239,25 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 }
 
 final ComponentLog logger = getLogger();
+final DocumentBuilderFactory 
docBuilderFactory=docBuilderFactoryRef.get();
+final DocumentBuilder docBuilder;
+try {
+docBuilder = docBuilderFactory.newDocumentBuilder();
+
+if (!context.getProperty(VALIDATE_DTD).asBoolean()) {
+docBuilder.setEntityResolver(new EntityResolver() {
+@Override
+public InputSource resolveEntity(String publicId, 
String systemId) throws SAXException, IOException {
+return new InputSource(new StringReader(""));
+}
+});
+}
+} catch (ParserConfigurationException e) {
+logger.error("Configuration error while constructing 
DocumentBuilder {}", new Object[]{e});
+session.transfer(flowFiles, REL_FAILURE);
+return;
--- End diff --

Throwing ProcessException would be more appropriate here, because the 
ParserConfigurationException is thrown regardless of incoming FlowFiles. By 
throwing ProcessException, incoming FlowFiles stay in the queue and can be 
processed after the user have fixed the issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133137160
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -277,8 +316,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 @Override
 public void process(final InputStream rawIn) throws 
IOException {
 try (final InputStream in = new 
BufferedInputStream(rawIn)) {
-final List rootList = (List) 
slashExpression.evaluate(new InputSource(in), NODESET);
-sourceRef.set(rootList.get(0));
+final Document document=docBuilder.parse(in);
--- End diff --

White space.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126957#comment-16126957
 ] 

ASF GitHub Bot commented on NIFI-4062:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133137160
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -277,8 +316,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 @Override
 public void process(final InputStream rawIn) throws 
IOException {
 try (final InputStream in = new 
BufferedInputStream(rawIn)) {
-final List rootList = (List) 
slashExpression.evaluate(new InputSource(in), NODESET);
-sourceRef.set(rootList.get(0));
+final Document document=docBuilder.parse(in);
--- End diff --

White space.


> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133135093
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -199,8 +216,11 @@ protected void init(final 
ProcessorInitializationContext context) {
 }
 
 @OnScheduled
-public void initializeXPathFactory() throws 
XPathFactoryConfigurationException {
+public void initializeFactory() throws 
XPathFactoryConfigurationException {
 
factoryRef.set(XPathFactory.newInstance(NamespaceConstant.OBJECT_MODEL_SAXON));
+final DocumentBuilderFactory 
docBuilderFactory=DocumentBuilderFactory.newInstance();
--- End diff --

Please add a white space before and after '='.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133133523
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -75,10 +45,47 @@
 import org.apache.nifi.processor.io.OutputStreamCallback;
 import org.apache.nifi.stream.io.BufferedInputStream;
 import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.w3c.dom.Document;
+import org.xml.sax.EntityResolver;
 import org.xml.sax.InputSource;
+import org.xml.sax.SAXException;
 
-import net.sf.saxon.lib.NamespaceConstant;
-import net.sf.saxon.xpath.XPathEvaluator;
+import javax.xml.namespace.QName;
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.ErrorListener;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Source;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerException;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.TransformerFactoryConfigurationError;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+import javax.xml.xpath.XPathExpression;
+import javax.xml.xpath.XPathExpressionException;
+import javax.xml.xpath.XPathFactory;
+import javax.xml.xpath.XPathFactoryConfigurationException;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.StringReader;
+import java.io.UnsupportedEncodingException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static javax.xml.xpath.XPathConstants.NODESET;
+import static javax.xml.xpath.XPathConstants.STRING;
--- End diff --

I assume these import statement changes are automatically generated by IDE. 
Is it possible to change only necessary lines of code?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133137297
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -309,18 +349,18 @@ public void process(final InputStream rawIn) throws 
IOException {
 }
 
 if (returnType == NODESET) {
-List nodeList = (List) result;
-if (nodeList.isEmpty()) {
+DOMNodeList nodeList = (DOMNodeList) result;
+if (nodeList.getLength()==0) {
--- End diff --

White space.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133135972
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXPath.java
 ---
@@ -125,6 +132,14 @@
 .defaultValue(RETURN_TYPE_AUTO)
 .build();
 
+public static final PropertyDescriptor VALIDATE_DTD = new 
PropertyDescriptor.Builder()
+.name("Validate DTD")
+.description("Specifies whether or not the XML content should 
be validated against the DTD.")
+.required(false)
--- End diff --

A select type property with a default value is usually defined as 
'required(true)'. If not, 'No value' can be selected from the pull down menu, 
but it will be overwritten by the default value. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2080: NIFI-4062 Provide an option to disable DTD validati...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2080#discussion_r133138260
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXQuery.java
 ---
@@ -272,10 +315,9 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 public void process(final InputStream rawIn) throws 
IOException {
 try (final InputStream in = new 
BufferedInputStream(rawIn)) {
 XQueryEvaluator qe = slashExpression.load();
-qe.setSource(new SAXSource(new InputSource(in)));
-DocumentBuilderFactory dfactory = 
DocumentBuilderFactory.newInstance();
-dfactory.setNamespaceAware(true);
-Document dom = 
dfactory.newDocumentBuilder().newDocument();
+final Document document=docBuilder.parse(in);
+qe.setSource(new DOMSource(document));
+Document dom = docBuilder.newDocument();
--- End diff --

Is there any performance impact by changing from SAX to DOM to evaluate a 
large XML document?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---