[jira] [Updated] (NIFI-5558) NiFi build on master fails due to decryption issue
[ https://issues.apache.org/jira/browse/NIFI-5558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-5558: Labels: encryption key-management properties security sideeffect test (was: ) Affects Version/s: 1.7.1 Status: Patch Available (was: In Progress) > NiFi build on master fails due to decryption issue > -- > > Key: NIFI-5558 > URL: https://issues.apache.org/jira/browse/NIFI-5558 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.7.1 > Environment: Apache Maven 3.5.0 > (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00) > Java version: 1.8.0_141, vendor: Oracle Corporation > Java home: > /Library/Java/JavaVirtualMachines/jdk1.8.0_141.jdk/Contents/Home/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac" >Reporter: Joseph Witt >Assignee: Andy LoPresto >Priority: Major > Labels: security, test, encryption, key-management, properties, > sideeffect > > ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.39 s > <<< FAILURE! - in > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest > [ERROR] > testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest) > Time elapsed: 1.318 s <<< ERROR! > org.apache.nifi.encrypt.EncryptionException: > org.apache.nifi.encrypt.EncryptionException: Could not decrypt sensitive value > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) > Caused by: org.apache.nifi.encrypt.EncryptionException: Could not decrypt > sensitive value > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) > Caused by: javax.crypto.BadPaddingException: pad block corrupted > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5558) NiFi build on master fails due to decryption issue
[
https://issues.apache.org/jira/browse/NIFI-5558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595873#comment-16595873
]
ASF GitHub Bot commented on NIFI-5558:
--
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2972
The issue that was encountered by some users was that the previous
mechanism for creating a test `NiFiProperties` instance was
`NiFiProperties.createBasicNiFiProperties(String filePath, Map
additionalProperties)`. When provided a `null` file path, the code tried to use
`System.getProperty("nifi.properties.file.path")` to locate the
`nifi.properties` file. On my machine, the test code which was evaluating the
_empty key_ scenario returned an empty `NiFiProperties` instance with only the
overridden properties populated, but on other machines, this code was loading
the properties from `src/test/resources/conf/nifi.properties` where the
`nifi.sensitive.props.key=key`. Because of this, the two passwords were not
equal, and the keys derived from them were different. When you attempt to
decrypt data with the wrong key, a `pad block corrupted` error is common.
The tests still work for me in the following scenarios, so I need a user
who encountered the error to validate that this fixes the issue on their
system.
* via IntelliJ Run/Debug
* via IntelliJ Maven execution
* via command-line Maven execution
* specific test using `mvn clean test
-Dtest=PopularVoteFlowElectionFactoryBeanTest`
* module using `.../nifi-framework-cluster $ mvn clean test`
* full build from root
* with JCE Unlimited Strength Jurisdiction Policies
* without JCE USJP
> NiFi build on master fails due to decryption issue
> --
>
> Key: NIFI-5558
> URL: https://issues.apache.org/jira/browse/NIFI-5558
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
> Environment: Apache Maven 3.5.0
> (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00)
> Java version: 1.8.0_141, vendor: Oracle Corporation
> Java home:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_141.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
>Reporter: Joseph Witt
>Assignee: Andy LoPresto
>Priority: Major
>
> ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.39 s
> <<< FAILURE! - in
> org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest
> [ERROR]
> testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest)
> Time elapsed: 1.318 s <<< ERROR!
> org.apache.nifi.encrypt.EncryptionException:
> org.apache.nifi.encrypt.EncryptionException: Could not decrypt sensitive value
> at
> org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82)
> Caused by: org.apache.nifi.encrypt.EncryptionException: Could not decrypt
> sensitive value
> at
> org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82)
> Caused by: javax.crypto.BadPaddingException: pad block corrupted
> at
> org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2972: NIFI-5558 Fixed unit test to avoid contamination from Syst...
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2972
The issue that was encountered by some users was that the previous
mechanism for creating a test `NiFiProperties` instance was
`NiFiProperties.createBasicNiFiProperties(String filePath, Map
additionalProperties)`. When provided a `null` file path, the code tried to use
`System.getProperty("nifi.properties.file.path")` to locate the
`nifi.properties` file. On my machine, the test code which was evaluating the
_empty key_ scenario returned an empty `NiFiProperties` instance with only the
overridden properties populated, but on other machines, this code was loading
the properties from `src/test/resources/conf/nifi.properties` where the
`nifi.sensitive.props.key=key`. Because of this, the two passwords were not
equal, and the keys derived from them were different. When you attempt to
decrypt data with the wrong key, a `pad block corrupted` error is common.
The tests still work for me in the following scenarios, so I need a user
who encountered the error to validate that this fixes the issue on their
system.
* via IntelliJ Run/Debug
* via IntelliJ Maven execution
* via command-line Maven execution
* specific test using `mvn clean test
-Dtest=PopularVoteFlowElectionFactoryBeanTest`
* module using `.../nifi-framework-cluster $ mvn clean test`
* full build from root
* with JCE Unlimited Strength Jurisdiction Policies
* without JCE USJP
---
[jira] [Commented] (NIFI-5558) NiFi build on master fails due to decryption issue
[ https://issues.apache.org/jira/browse/NIFI-5558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595869#comment-16595869 ] ASF GitHub Bot commented on NIFI-5558: -- GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/2972 NIFI-5558 Fixed unit test to avoid contamination from System property… … "nifi.properties.file.path" when creating test NiFiProperties instance. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5558 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2972.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2972 commit 9c9db43bd157d84ff58e9003f646c4ff38147f56 Author: Andy LoPresto Date: 2018-08-29T02:53:45Z NIFI-5558 Fixed unit test to avoid contamination from System property "nifi.properties.file.path" when creating test NiFiProperties instance. > NiFi build on master fails due to decryption issue > -- > > Key: NIFI-5558 > URL: https://issues.apache.org/jira/browse/NIFI-5558 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build > Environment: Apache Maven 3.5.0 > (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00) > Java version: 1.8.0_141, vendor: Oracle Corporation > Java home: > /Library/Java/JavaVirtualMachines/jdk1.8.0_141.jdk/Contents/Home/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac" >Reporter: Joseph Witt >Assignee: Andy LoPresto >Priority: Major > > ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.39 s > <<< FAILURE! - in > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest > [ERROR] > testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest) > Time elapsed: 1.318 s <<< ERROR! > org.apache.nifi.encrypt.EncryptionException: > org.apache.nifi.encrypt.EncryptionException: Could not decrypt sensitive value > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) > Caused by: org.apache.nifi.encrypt.EncryptionException: Could not decrypt > sensitive value > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) > Caused by: javax.crypto.BadPaddingException: pad block corrupted > at >
[GitHub] nifi pull request #2972: NIFI-5558 Fixed unit test to avoid contamination fr...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/2972 NIFI-5558 Fixed unit test to avoid contamination from System property⦠⦠"nifi.properties.file.path" when creating test NiFiProperties instance. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5558 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2972.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2972 commit 9c9db43bd157d84ff58e9003f646c4ff38147f56 Author: Andy LoPresto Date: 2018-08-29T02:53:45Z NIFI-5558 Fixed unit test to avoid contamination from System property "nifi.properties.file.path" when creating test NiFiProperties instance. ---
[jira] [Commented] (NIFI-1893) Add processor for validating JSON
[
https://issues.apache.org/jira/browse/NIFI-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595846#comment-16595846
]
HondaWei commented on NIFI-1893:
Hi
I would like to use the following library to implement this processor
[https://github.com/java-json-tools/json-schema-validator]
I have already finished it, may someone assign this issue to me?
thank you!
> Add processor for validating JSON
> -
>
> Key: NIFI-1893
> URL: https://issues.apache.org/jira/browse/NIFI-1893
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Matt Burgess
>Priority: Major
>
> NiFi has a ValidateXml processor to validate incoming XML files against a
> schema. It would be good to have one to validate JSON files as well.
> For example, an input JSON of:
> {
> name: "Test",
> timestamp: 1463499695,
> tags: {
>"host": "Test_1",
>"ip" : "1.1.1.1"
> },
> fields: {
> "cpu": 10.2,
> "load": 15.6
> }
> }
> Could be validated successfully against the following "schema":
> {
> "type": "object",
> "required": ["name", "tags", "timestamp", "fields"],
> "properties": {
> "name": {"type": "string"},
> "timestamp": {"type": "integer"},
> "tags": {"type": "object", "items": {"type": "string"}},
> "fields": { "type": "object"}
> }
> }
> There is at least one ASF-friendly library that could be used for
> implementation: https://github.com/everit-org/json-schema
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-4426) Remove custom jBCrypt implementation because Java 7 is no longer supported
[
https://issues.apache.org/jira/browse/NIFI-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough updated NIFI-4426:
---
Attachment: (was: Screen Shot 2018-08-28 at 6.38.31 PM.png)
> Remove custom jBCrypt implementation because Java 7 is no longer supported
> --
>
> Key: NIFI-4426
> URL: https://issues.apache.org/jira/browse/NIFI-4426
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Minor
> Labels: security
> Attachments: Screen Shot 2018-08-28 at 6.38.16 PM.png, Screen Shot
> 2018-08-28 at 6.38.31 PM.png
>
>
> The {{jBCrypt}} library is included and slightly modified in order to provide
> Java 7 compatibility because the external module is compiled for Java 8. Now
> that NiFi doesn't support Java 7, this modification can be removed and the
> standalone module can be depended upon via Maven as per normal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4426) Remove custom jBCrypt implementation because Java 7 is no longer supported
[
https://issues.apache.org/jira/browse/NIFI-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595677#comment-16595677
]
Nathan Gough commented on NIFI-4426:
Minor differences between
nifi-security-utils/src/main/java/org/apache/nifi/security/util/crypto/bcrypt/BCrypt.java
and org.mindrot.jbcrypt.BCrypt:
!Screen Shot 2018-08-28 at 6.38.16 PM.png!
!Screen Shot 2018-08-28 at 6.38.31 PM.png!
> Remove custom jBCrypt implementation because Java 7 is no longer supported
> --
>
> Key: NIFI-4426
> URL: https://issues.apache.org/jira/browse/NIFI-4426
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Minor
> Labels: security
> Attachments: Screen Shot 2018-08-28 at 6.38.16 PM.png, Screen Shot
> 2018-08-28 at 6.38.31 PM.png
>
>
> The {{jBCrypt}} library is included and slightly modified in order to provide
> Java 7 compatibility because the external module is compiled for Java 8. Now
> that NiFi doesn't support Java 7, this modification can be removed and the
> standalone module can be depended upon via Maven as per normal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-4426) Remove custom jBCrypt implementation because Java 7 is no longer supported
[
https://issues.apache.org/jira/browse/NIFI-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough updated NIFI-4426:
---
Attachment: Screen Shot 2018-08-28 at 6.38.31 PM.png
> Remove custom jBCrypt implementation because Java 7 is no longer supported
> --
>
> Key: NIFI-4426
> URL: https://issues.apache.org/jira/browse/NIFI-4426
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Minor
> Labels: security
> Attachments: Screen Shot 2018-08-28 at 6.38.16 PM.png, Screen Shot
> 2018-08-28 at 6.38.31 PM.png
>
>
> The {{jBCrypt}} library is included and slightly modified in order to provide
> Java 7 compatibility because the external module is compiled for Java 8. Now
> that NiFi doesn't support Java 7, this modification can be removed and the
> standalone module can be depended upon via Maven as per normal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-4426) Remove custom jBCrypt implementation because Java 7 is no longer supported
[
https://issues.apache.org/jira/browse/NIFI-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough updated NIFI-4426:
---
Attachment: Screen Shot 2018-08-28 at 6.38.31 PM.png
Screen Shot 2018-08-28 at 6.38.16 PM.png
> Remove custom jBCrypt implementation because Java 7 is no longer supported
> --
>
> Key: NIFI-4426
> URL: https://issues.apache.org/jira/browse/NIFI-4426
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Minor
> Labels: security
> Attachments: Screen Shot 2018-08-28 at 6.38.16 PM.png, Screen Shot
> 2018-08-28 at 6.38.31 PM.png
>
>
> The {{jBCrypt}} library is included and slightly modified in order to provide
> Java 7 compatibility because the external module is compiled for Java 8. Now
> that NiFi doesn't support Java 7, this modification can be removed and the
> standalone module can be depended upon via Maven as per normal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4426) Remove custom jBCrypt implementation because Java 7 is no longer supported
[
https://issues.apache.org/jira/browse/NIFI-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595675#comment-16595675
]
Nathan Gough commented on NIFI-4426:
Minor differences between
nifi-security-utils/src/main/java/org/apache/nifi/security/util/crypto/bcrypt/BCrypt.java
and org.mindrot.jbcrypt.BCrypt:
!Screen Shot 2018-08-28 at 6.38.16 PM.png!
and
!Screen Shot 2018-08-28 at 6.38.31 PM.png!
> Remove custom jBCrypt implementation because Java 7 is no longer supported
> --
>
> Key: NIFI-4426
> URL: https://issues.apache.org/jira/browse/NIFI-4426
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Minor
> Labels: security
>
> The {{jBCrypt}} library is included and slightly modified in order to provide
> Java 7 compatibility because the external module is compiled for Java 8. Now
> that NiFi doesn't support Java 7, this modification can be removed and the
> standalone module can be depended upon via Maven as per normal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4426) Remove custom jBCrypt implementation because Java 7 is no longer supported
[
https://issues.apache.org/jira/browse/NIFI-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595646#comment-16595646
]
Nathan Gough commented on NIFI-4426:
Checked if there was any difference between:
{code:java}
org.mindrot
jbcrypt
0.4
{code}
and
{noformat}
de.svenkubiak
jBCrypt
0.4.1
{noformat}
but I found these to be identical code.
> Remove custom jBCrypt implementation because Java 7 is no longer supported
> --
>
> Key: NIFI-4426
> URL: https://issues.apache.org/jira/browse/NIFI-4426
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Minor
> Labels: security
>
> The {{jBCrypt}} library is included and slightly modified in order to provide
> Java 7 compatibility because the external module is compiled for Java 8. Now
> that NiFi doesn't support Java 7, this modification can be removed and the
> standalone module can be depended upon via Maven as per normal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-601) Add scheduling information to manifest
[ https://issues.apache.org/jira/browse/MINIFICPP-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFICPP-601. --- Resolution: Fixed > Add scheduling information to manifest > -- > > Key: MINIFICPP-601 > URL: https://issues.apache.org/jira/browse/MINIFICPP-601 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Add scheduling information to manifest -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-600) Add default value to manifest
[ https://issues.apache.org/jira/browse/MINIFICPP-600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFICPP-600. --- Resolution: Fixed Fix Version/s: 0.6.0 > Add default value to manifest > - > > Key: MINIFICPP-600 > URL: https://issues.apache.org/jira/browse/MINIFICPP-600 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Add default value to manifest -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #393: MINIFICPP 601: Add Scheduling information
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/393 ---
[jira] [Created] (NIFI-5559) Allow usage of Trusted Hostname on all HTTP-accessing processors
Colin Dean created NIFI-5559:
Summary: Allow usage of Trusted Hostname on all HTTP-accessing
processors
Key: NIFI-5559
URL: https://issues.apache.org/jira/browse/NIFI-5559
Project: Apache NiFi
Issue Type: Wish
Components: Core Framework
Affects Versions: 1.7.1
Reporter: Colin Dean
I've used several PutHTTP and GetHTTP processors in a workflow for development
before realizing that I cannot configure a {{Trusted Hostname}} property for
those like I can for InvokeHTTP. Configuration and setup limitations of the
SSLContextService have me wanting to avoid it at all costs.
I think the workaround is to use InvokeHTTP in place of the others but it'd be
nice to be able to set a trusted hostname anywhere that an SSLContentService
can be used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #393: MINIFICPP 601: Add Scheduling information
Github user apiri commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/393#discussion_r213437908 --- Diff: bootstrap.sh --- @@ -25,6 +25,7 @@ CORES=1 BUILD="false" PACKAGE="false" BUILD_IDENTIFIER="" +BUILD_IDENTIFIER="" --- End diff -- Will clean this up on merge ---
[jira] [Commented] (MINIFICPP-592) Restful lookups in RPG should be bypassed when cURL is disabled.
[ https://issues.apache.org/jira/browse/MINIFICPP-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595537#comment-16595537 ] ASF GitHub Bot commented on MINIFICPP-592: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/389 > Restful lookups in RPG should be bypassed when cURL is disabled. > > > Key: MINIFICPP-592 > URL: https://issues.apache.org/jira/browse/MINIFICPP-592 > Project: NiFi MiNiFi C++ > Issue Type: Test >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Changes to the RPG that performed a lookup on the NiFi Rest API should be > bypassed and the YAML config used when cURL support is disabled in a client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-592) Restful lookups in RPG should be bypassed when cURL is disabled.
[ https://issues.apache.org/jira/browse/MINIFICPP-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFICPP-592. --- Resolution: Fixed Fix Version/s: 0.6.0 > Restful lookups in RPG should be bypassed when cURL is disabled. > > > Key: MINIFICPP-592 > URL: https://issues.apache.org/jira/browse/MINIFICPP-592 > Project: NiFi MiNiFi C++ > Issue Type: Test >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Changes to the RPG that performed a lookup on the NiFi Rest API should be > bypassed and the YAML config used when cURL support is disabled in a client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #389: MINIFICPP-592: Update RPG to fall back wh...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/389 ---
[jira] [Updated] (NIFI-5060) UpdateRecord substringAfter and substringAfterLast only increments by 1
[
https://issues.apache.org/jira/browse/NIFI-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Moser updated NIFI-5060:
Resolution: Fixed
Fix Version/s: 1.7.0
Status: Resolved (was: Patch Available)
> UpdateRecord substringAfter and substringAfterLast only increments by 1
> ---
>
> Key: NIFI-5060
> URL: https://issues.apache.org/jira/browse/NIFI-5060
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Green
>Priority: Major
> Labels: easyfix, newbie
> Fix For: 1.7.0
>
> Attachments: Validate_substringafter_Behavior.xml
>
>
> This is my first submitted issue, so please feel free to point me in the
> correct direction if I make process mistakes.
> Replication:
> Drag a GenerateFlowFile onto the canvas and configure this property, and set
> run schedule to some high value like 600 seconds
> "Custom Text" \{"value": "01230123"}
> Connect GenerateFlowFile with an UpdateAttribute set to add the attribute
> "avro.schema" with a value of
>
> {code:java}
> {
> "type": "record",
> "name": "test",
> "fields" : [{"name": "value", "type": "string"}]
> }
> {code}
>
>
> Connect UpdateAttribute to an UpdateRecord onto the canvas, Autoterminate
> success and failure. Set the Record Reader to a new JSONTreeReader. On the
> JsonTreeReader configure it to use the "Use 'Schema Text' Attribute".
> Create a JsonRecordSetWriter and set the Schema Text to:
>
>
> {code:java}
> {
> "type": "record",
> "name": "test",
> "fields" : [
> {"name": "value", "type": "string"},
> {"name": "example1", "type": "string"},
> {"name": "example2", "type": "string"},
> {"name": "example3", "type": "string"},
> {"name": "example4", "type": "string"}
> ]
> }
> {code}
>
> Add the following properties to UpdateRecord
>
> ||Heading 1||Heading 2||
> |/example1|substringAfter(/value, "1") |
> |/example2|substringAfter(/value, "123") |
> |/example3|substringAfterLast(/value, "1")|
> |/example4|substringAfterLast(/value, "123")|
>
> Resulting record currently:
>
> {code:java}
> [{
> "value" : "01230123",
> "example1" : "230123",
> "example2" : "30123",
> "example3" : "23",
> "example4" : "3"
> }]
> {code}
>
>
>
> Problem:
> When using the UpdateRecord processor, and the substringAfter() function
> after the search phrase is found it will only increment the substring
> returned by 1 rather than the length of the search term.
> Based off XPath and other implementations of substringAfter functions I've
> seen the value returned should remove the search term rather than just the
> first character of the search term.
>
>
> Resulting record should be:
>
> {code:java}
> [{
> "value" : "01230123",
> "example1" : "230123",
> "example2" : "0123",
> "example3" : "23",
> "example4" : ""
> }]
> {code}
>
>
> I'm cleaning up a fix with test code that will change the increment from 1 to
> the length of the search terms.
> It appears substringBefore are not impacted by the behavior as always returns
> the index before the found search term which is the expected behavior
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5542) Add support for node groups to FileAccessPolicyProvider
[
https://issues.apache.org/jira/browse/NIFI-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595487#comment-16595487
]
ASF GitHub Bot commented on NIFI-5542:
--
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2970#discussion_r213438246
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-file-authorizer/src/main/java/org/apache/nifi/authorization/FileAccessPolicyProvider.java
---
@@ -114,6 +114,7 @@ private static JAXBContext initializeJaxbContext(final
String contextPath) {
static final String WRITE_CODE = "W";
static final String PROP_NODE_IDENTITY_PREFIX = "Node Identity ";
+static final String PROP_NODE_GROUP = "Node Group";
--- End diff --
The [Admin
Guide](https://github.com/apache/nifi/blob/master/nifi-docs/src/main/asciidoc/administration-guide.adoc#fileaccesspolicyprovider)
and default
[authorizers.xml](https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/authorizers.xml)
file should be updated to include this property and its usage.
> Add support for node groups to FileAccessPolicyProvider
> ---
>
> Key: NIFI-5542
> URL: https://issues.apache.org/jira/browse/NIFI-5542
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Andrew Christianson
>Priority: Major
>
> Currently in FileAccessPolicyProvider, it is possible to specify a set of
> node identities, which are given access to /proxy. This works well for static
> clusters, but does not work so well for dynamic clusters (scaling up/down #
> of nodes) because we don't know in advance what the node identities will be
> or how many there will be.
> In order to support dynamic sets of node identities, add support for
> specifying a "Node Group," for which all identities in the group will be
> granted access to /proxy. A UserGroupProvider can then be implemented to
> gather node identities dynamically from the cluster environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2970: NIFI-5542 Added support for node groups to FileAcce...
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2970#discussion_r213438246
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-file-authorizer/src/main/java/org/apache/nifi/authorization/FileAccessPolicyProvider.java
---
@@ -114,6 +114,7 @@ private static JAXBContext initializeJaxbContext(final
String contextPath) {
static final String WRITE_CODE = "W";
static final String PROP_NODE_IDENTITY_PREFIX = "Node Identity ";
+static final String PROP_NODE_GROUP = "Node Group";
--- End diff --
The [Admin
Guide](https://github.com/apache/nifi/blob/master/nifi-docs/src/main/asciidoc/administration-guide.adoc#fileaccesspolicyprovider)
and default
[authorizers.xml](https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/authorizers.xml)
file should be updated to include this property and its usage.
---
[jira] [Commented] (MINIFICPP-592) Restful lookups in RPG should be bypassed when cURL is disabled.
[
https://issues.apache.org/jira/browse/MINIFICPP-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595474#comment-16595474
]
ASF GitHub Bot commented on MINIFICPP-592:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/389#discussion_r213438194
--- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
@@ -209,7 +225,8 @@ void RemoteProcessorGroupPort::onTrigger(const
std::shared_ptrlog_trace("On trigger %s", getUUIDStr());
- if (url_.empty()) {
+ /*
--- End diff --
that's fine by me
> Restful lookups in RPG should be bypassed when cURL is disabled.
>
>
> Key: MINIFICPP-592
> URL: https://issues.apache.org/jira/browse/MINIFICPP-592
> Project: NiFi MiNiFi C++
> Issue Type: Test
>Reporter: Mr TheSegfault
>Assignee: Mr TheSegfault
>Priority: Major
>
> Changes to the RPG that performed a lookup on the NiFi Rest API should be
> bypassed and the YAML config used when cURL support is disabled in a client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #389: MINIFICPP-592: Update RPG to fall back wh...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/389#discussion_r213438194
--- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
@@ -209,7 +225,8 @@ void RemoteProcessorGroupPort::onTrigger(const
std::shared_ptrlog_trace("On trigger %s", getUUIDStr());
- if (url_.empty()) {
+ /*
--- End diff --
that's fine by me
---
[jira] [Commented] (MINIFICPP-592) Restful lookups in RPG should be bypassed when cURL is disabled.
[
https://issues.apache.org/jira/browse/MINIFICPP-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595461#comment-16595461
]
ASF GitHub Bot commented on MINIFICPP-592:
--
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/389#discussion_r213437719
--- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
@@ -209,7 +225,8 @@ void RemoteProcessorGroupPort::onTrigger(const
std::shared_ptrlog_trace("On trigger %s", getUUIDStr());
- if (url_.empty()) {
+ /*
--- End diff --
looks like this is handled elsewhere with your changes. I can clean this
up on merge if that works for you.
> Restful lookups in RPG should be bypassed when cURL is disabled.
>
>
> Key: MINIFICPP-592
> URL: https://issues.apache.org/jira/browse/MINIFICPP-592
> Project: NiFi MiNiFi C++
> Issue Type: Test
>Reporter: Mr TheSegfault
>Assignee: Mr TheSegfault
>Priority: Major
>
> Changes to the RPG that performed a lookup on the NiFi Rest API should be
> bypassed and the YAML config used when cURL support is disabled in a client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #389: MINIFICPP-592: Update RPG to fall back wh...
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/389#discussion_r213437719
--- Diff: libminifi/src/RemoteProcessorGroupPort.cpp ---
@@ -209,7 +225,8 @@ void RemoteProcessorGroupPort::onTrigger(const
std::shared_ptrlog_trace("On trigger %s", getUUIDStr());
- if (url_.empty()) {
+ /*
--- End diff --
looks like this is handled elsewhere with your changes. I can clean this
up on merge if that works for you.
---
[jira] [Commented] (NIFI-5456) PutKinesisStream - Fails to work with AWS Private Link endpoint
[
https://issues.apache.org/jira/browse/NIFI-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595433#comment-16595433
]
ASF GitHub Bot commented on NIFI-5456:
--
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2968#discussion_r213431435
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
---
@@ -286,7 +286,7 @@ protected void
initializeRegionAndEndpoint(ProcessContext context) {
final String urlstr =
StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue());
if (!urlstr.isEmpty()) {
getLogger().info("Overriding endpoint with {}", new
Object[]{urlstr});
-this.client.setEndpoint(urlstr);
+this.client.setEndpoint(urlstr,
this.client.getServiceName(), this.region.getName());
--- End diff --
these values will always be present? We dont have to guard from them being
null/undefined/different than the API being called expects?
> PutKinesisStream - Fails to work with AWS Private Link endpoint
> ---
>
> Key: NIFI-5456
> URL: https://issues.apache.org/jira/browse/NIFI-5456
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.6.0, 1.7.1
> Environment: RedHat 6
>Reporter: Ariel Godinez
>Assignee: Sivaprasanna Sethuraman
>Priority: Major
> Labels: easyfix
> Attachments:
> 0001-NIFI-5456-AWS-clients-now-work-with-private-link-end.patch
>
>
> NiFi version: 1.6.0
> PutKinesisStream fails to put due to invalid signing information when using
> an AWS Private Link as the endpoint override URL. The endpoint override URL
> pattern for private links is like below along with the error that NiFi
> outputs when we attempt to use this type of URL as the 'Endpoint Override
> URL' property value.
> Endpoint Override URL:
> [https://vpce-|https://vpce-/].kinesis.us-east-2.vpce.amazonaws.com
> ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream"
> PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish
> due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException:
> Credential should be scoped to a valid region, not 'vpce'. (Service:
> AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException;
> Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles
> [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1532469012962-1,
> container=content002, section=1], offset=2159750,
> length=534625],offset=0,name=900966573101260,size=534625]]
>
> It looks like 'vpce' is being extracted from the url as the region name when
> it should be getting 'us-east-2'. We were able to get this processor to work
> correctly by explicitly passing in the region and service using
> 'setEndpoint(String endpoint, String serviceName, String regionId)' instead
> of 'setEndpoint(String endpoint)' in
> 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java'
> line 289
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2968: NIFI-5456: AWS clients now work with private link e...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2968#discussion_r213431435
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
---
@@ -286,7 +286,7 @@ protected void
initializeRegionAndEndpoint(ProcessContext context) {
final String urlstr =
StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue());
if (!urlstr.isEmpty()) {
getLogger().info("Overriding endpoint with {}", new
Object[]{urlstr});
-this.client.setEndpoint(urlstr);
+this.client.setEndpoint(urlstr,
this.client.getServiceName(), this.region.getName());
--- End diff --
these values will always be present? We dont have to guard from them being
null/undefined/different than the API being called expects?
---
[jira] [Commented] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595422#comment-16595422 ] ASF GitHub Bot commented on NIFI-5479: -- Github user joewitt closed the pull request at: https://github.com/apache/nifi/pull/2933 > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Major > Fix For: 1.8.0 > > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2933: NIFI-5479 Upgraded Jetty. Moved where we unpack bun...
Github user joewitt closed the pull request at: https://github.com/apache/nifi/pull/2933 ---
[jira] [Commented] (NIFI-5353) Add a JoltTransformRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595416#comment-16595416 ] ASF GitHub Bot commented on NIFI-5353: -- Github user mattyb149 closed the pull request at: https://github.com/apache/nifi/pull/2833 > Add a JoltTransformRecord processor > --- > > Key: NIFI-5353 > URL: https://issues.apache.org/jira/browse/NIFI-5353 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.8.0 > > > JOLT is a great Domain-Specific Language (DSL) and engine for transforming > JSON documents, which our JoltTransformJSON processor leverages. However, the > JOLT engine simply translates the JSON text into a Java object, runs the > transformation based on the spec, and then transforms the object(s) back into > JSON. This is very similar to how our record-based processors work, and JOLT > itself is not bound to JSON per se. > We can leverage JOLT to provide a record-based processor to run JOLT > transformations on any incoming record sets, such as XML, CSV, Avro, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2833: NIFI-5353: Add JoltTransformRecord processor
Github user mattyb149 closed the pull request at: https://github.com/apache/nifi/pull/2833 ---
[jira] [Commented] (NIFI-5353) Add a JoltTransformRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595412#comment-16595412 ] ASF GitHub Bot commented on NIFI-5353: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2833 @mattyb149 thanks for the updates! This is looking much better now. Was able to run through some simple test cases and everything appears to work as expected. +1 merged to master! > Add a JoltTransformRecord processor > --- > > Key: NIFI-5353 > URL: https://issues.apache.org/jira/browse/NIFI-5353 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.8.0 > > > JOLT is a great Domain-Specific Language (DSL) and engine for transforming > JSON documents, which our JoltTransformJSON processor leverages. However, the > JOLT engine simply translates the JSON text into a Java object, runs the > transformation based on the spec, and then transforms the object(s) back into > JSON. This is very similar to how our record-based processors work, and JOLT > itself is not bound to JSON per se. > We can leverage JOLT to provide a record-based processor to run JOLT > transformations on any incoming record sets, such as XML, CSV, Avro, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5353) Add a JoltTransformRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-5353: - Resolution: Fixed Fix Version/s: 1.8.0 Status: Resolved (was: Patch Available) > Add a JoltTransformRecord processor > --- > > Key: NIFI-5353 > URL: https://issues.apache.org/jira/browse/NIFI-5353 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.8.0 > > > JOLT is a great Domain-Specific Language (DSL) and engine for transforming > JSON documents, which our JoltTransformJSON processor leverages. However, the > JOLT engine simply translates the JSON text into a Java object, runs the > transformation based on the spec, and then transforms the object(s) back into > JSON. This is very similar to how our record-based processors work, and JOLT > itself is not bound to JSON per se. > We can leverage JOLT to provide a record-based processor to run JOLT > transformations on any incoming record sets, such as XML, CSV, Avro, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2833: NIFI-5353: Add JoltTransformRecord processor
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2833 @mattyb149 thanks for the updates! This is looking much better now. Was able to run through some simple test cases and everything appears to work as expected. +1 merged to master! ---
[jira] [Commented] (NIFI-5353) Add a JoltTransformRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595409#comment-16595409 ] ASF subversion and git services commented on NIFI-5353: --- Commit 53969adcc8a68185447b4e3546d634663e74c6f0 in nifi's branch refs/heads/master from [~ca9mbu] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=53969ad ] NIFI-5353: Add JoltTransformRecord processor > Add a JoltTransformRecord processor > --- > > Key: NIFI-5353 > URL: https://issues.apache.org/jira/browse/NIFI-5353 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > JOLT is a great Domain-Specific Language (DSL) and engine for transforming > JSON documents, which our JoltTransformJSON processor leverages. However, the > JOLT engine simply translates the JSON text into a Java object, runs the > transformation based on the spec, and then transforms the object(s) back into > JSON. This is very similar to how our record-based processors work, and JOLT > itself is not bound to JSON per se. > We can leverage JOLT to provide a record-based processor to run JOLT > transformations on any incoming record sets, such as XML, CSV, Avro, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-598) Add supported relationships to agent manifest
[ https://issues.apache.org/jira/browse/MINIFICPP-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-598: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Add supported relationships to agent manifest > - > > Key: MINIFICPP-598 > URL: https://issues.apache.org/jira/browse/MINIFICPP-598 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Probably need to include the relationship name and description > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #393: MINIFICPP 601: Add Scheduling information
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/393 @phrocker would you mind please rebasing this on master? I unfortunately missed the PR coming when reviewing #392. The merge conflicts look relatively straightforward but I don't want to overlook something in that process ---
[jira] [Commented] (MINIFICPP-598) Add supported relationships to agent manifest
[ https://issues.apache.org/jira/browse/MINIFICPP-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595311#comment-16595311 ] ASF GitHub Bot commented on MINIFICPP-598: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/392 > Add supported relationships to agent manifest > - > > Key: MINIFICPP-598 > URL: https://issues.apache.org/jira/browse/MINIFICPP-598 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Probably need to include the relationship name and description > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #392: MINIFICPP-598: Add relationship support i...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/392 ---
[jira] [Commented] (NIFI-5558) NiFi build on master fails due to decryption issue
[ https://issues.apache.org/jira/browse/NIFI-5558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595273#comment-16595273 ] Andy LoPresto commented on NIFI-5558: - I am guessing this comes from not having the JCE Unlimited Strength Jurisdiction policies installed. I will reproduce and if so, rewrite the tests to comply with the limited strength crypto policy. > NiFi build on master fails due to decryption issue > -- > > Key: NIFI-5558 > URL: https://issues.apache.org/jira/browse/NIFI-5558 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build > Environment: Apache Maven 3.5.0 > (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00) > Java version: 1.8.0_141, vendor: Oracle Corporation > Java home: > /Library/Java/JavaVirtualMachines/jdk1.8.0_141.jdk/Contents/Home/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac" >Reporter: Joseph Witt >Priority: Major > > ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.39 s > <<< FAILURE! - in > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest > [ERROR] > testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest) > Time elapsed: 1.318 s <<< ERROR! > org.apache.nifi.encrypt.EncryptionException: > org.apache.nifi.encrypt.EncryptionException: Could not decrypt sensitive value > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) > Caused by: org.apache.nifi.encrypt.EncryptionException: Could not decrypt > sensitive value > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) > Caused by: javax.crypto.BadPaddingException: pad block corrupted > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5558) NiFi build on master fails due to decryption issue
[ https://issues.apache.org/jira/browse/NIFI-5558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto reassigned NIFI-5558: --- Assignee: Andy LoPresto > NiFi build on master fails due to decryption issue > -- > > Key: NIFI-5558 > URL: https://issues.apache.org/jira/browse/NIFI-5558 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build > Environment: Apache Maven 3.5.0 > (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00) > Java version: 1.8.0_141, vendor: Oracle Corporation > Java home: > /Library/Java/JavaVirtualMachines/jdk1.8.0_141.jdk/Contents/Home/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac" >Reporter: Joseph Witt >Assignee: Andy LoPresto >Priority: Major > > ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.39 s > <<< FAILURE! - in > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest > [ERROR] > testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest) > Time elapsed: 1.318 s <<< ERROR! > org.apache.nifi.encrypt.EncryptionException: > org.apache.nifi.encrypt.EncryptionException: Could not decrypt sensitive value > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) > Caused by: org.apache.nifi.encrypt.EncryptionException: Could not decrypt > sensitive value > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) > Caused by: javax.crypto.BadPaddingException: pad block corrupted > at > org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-598) Add supported relationships to agent manifest
[ https://issues.apache.org/jira/browse/MINIFICPP-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-598: -- Status: Patch Available (was: Open) > Add supported relationships to agent manifest > - > > Key: MINIFICPP-598 > URL: https://issues.apache.org/jira/browse/MINIFICPP-598 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Probably need to include the relationship name and description > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5558) NiFi build on master fails due to decryption issue
Joseph Witt created NIFI-5558: - Summary: NiFi build on master fails due to decryption issue Key: NIFI-5558 URL: https://issues.apache.org/jira/browse/NIFI-5558 Project: Apache NiFi Issue Type: Bug Components: Tools and Build Environment: Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00) Java version: 1.8.0_141, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_141.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac" Reporter: Joseph Witt ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.39 s <<< FAILURE! - in org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest [ERROR] testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest) Time elapsed: 1.318 s <<< ERROR! org.apache.nifi.encrypt.EncryptionException: org.apache.nifi.encrypt.EncryptionException: Could not decrypt sensitive value at org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) Caused by: org.apache.nifi.encrypt.EncryptionException: Could not decrypt sensitive value at org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) Caused by: javax.crypto.BadPaddingException: pad block corrupted at org.apache.nifi.cluster.coordination.flow.PopularVoteFlowElectionFactoryBeanTest.testGetObjectShouldPopulateDefaultSensitivePropsKeyIfEmpty(PopularVoteFlowElectionFactoryBeanTest.groovy:82) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #393: MINIFICPP 601: Add Scheduling information
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/393 MINIFICPP 601: Add Scheduling information This is predicated upon #392 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-601 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/393.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #393 commit 7ac309ba76813e5dbdb1e1857da4c6115a4c2fd1 Author: Marc Parisi Date: 2018-08-20T16:26:14Z MINIFICPP-598: Add relationship support into manifest MINIFICPP-600: Add default value into property descriptors commit 30a19a1c73c244352178b165acc7d29728df881a Author: Marc Parisi Date: 2018-08-22T00:22:57Z MINIFICPP-601: Add scheduling information ---
[GitHub] nifi pull request #2969: NIFI-5495 Made date format configurable.
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2969#discussion_r213383530 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java --- @@ -173,6 +175,29 @@ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .build(); --- End diff -- When I run the tests in intellij ( I did do a PR, but there was too much shrapnel ), the tests that compare the content to the known values fail, because the json is pretty printed in the result but not the expected. ---
[jira] [Commented] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595236#comment-16595236
]
ASF GitHub Bot commented on NIFI-5495:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2969#discussion_r213383530
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -173,6 +175,29 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
--- End diff --
When I run the tests in intellij ( I did do a PR, but there was too much
shrapnel ), the tests that compare the content to the known values fail,
because the json is pretty printed in the result but not the expected.
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Assignee: Mike Thomsen
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5557) PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
[
https://issues.apache.org/jira/browse/NIFI-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595228#comment-16595228
]
ASF GitHub Bot commented on NIFI-5557:
--
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/2971
@ekovacs thanks for your contribution. Are we sure there is no other
course for us than to parse the exception cause for this case? If not, that is
unfortunate but ok. The other question then is if this is a particular case we
need the HDFS client to tell us have we asked the Hadoop community to improve
this or filed a JIRA to that effect? If not we definitely should so that we
can check this in the future in a way that is safer/more reliable. Thanks!
> PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
> -
>
> Key: NIFI-5557
> URL: https://issues.apache.org/jira/browse/NIFI-5557
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Endre Kovacs
>Assignee: Endre Kovacs
>Priority: Major
>
> when using *PutHDFS* processor in a kerberized environment, with a flow
> "traffic" which approximately matches or less frequent then the lifetime of
> the ticket of the principal, we see this in the log:
> {code:java}
> INFO [Timer-Driven Process Thread-4] o.a.h.io.retry.RetryInvocationHandler
> Exception while invoking getFileInfo of class
> ClientNamenodeProtocolTranslatorPB over host2/ip2:8020 after 13 fail over
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for princi...@example.com to host2.example.com/ip2:8020;
> Host Details : local host is: "host1.example.com/ip1"; destination host is:
> "host2.example.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> at sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy135.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:254)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
> at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:222)
> {code}
> and the flowfile is routed to failure relationship.
> *To reproduce:*
> Create a principal in your KDC with two minutes ticket lifetime,
> and set up a similar flow:
> {code:java}
> GetFile => putHDFS - success- -> logAttributes
> \
> fail
>\
> -> logAttributes
> {code}
> copy a file to the input directory of the getFile processor. If the influx
> of the flowfile is much more frequent, then the expiration time of the ticket:
> {code:java}
> watch -n 5 "cp book.txt /path/to/input"
> {code}
> then the flow will successfully run without issue.
> If we adjust this, to:
> {code:java}
> watch -n 121 "cp book.txt /path/to/input"
> {code}
> then we will observe this issue.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2971: NIFI-5557: handling expired ticket by rollback and penaliz...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2971 @ekovacs thanks for your contribution. Are we sure there is no other course for us than to parse the exception cause for this case? If not, that is unfortunate but ok. The other question then is if this is a particular case we need the HDFS client to tell us have we asked the Hadoop community to improve this or filed a JIRA to that effect? If not we definitely should so that we can check this in the future in a way that is safer/more reliable. Thanks! ---
[jira] [Commented] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595210#comment-16595210
]
ASF GitHub Bot commented on NIFI-5495:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2969#discussion_r213375336
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -173,6 +175,29 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
--- End diff --
I didn't see any failures when I ran. What are you seeing?
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Assignee: Mike Thomsen
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2969: NIFI-5495 Made date format configurable.
Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/2969#discussion_r213375336 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java --- @@ -173,6 +175,29 @@ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .build(); --- End diff -- I didn't see any failures when I ran. What are you seeing? ---
[jira] [Commented] (NIFI-5557) PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
[
https://issues.apache.org/jira/browse/NIFI-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595158#comment-16595158
]
ASF GitHub Bot commented on NIFI-5557:
--
GitHub user ekovacs opened a pull request:
https://github.com/apache/nifi/pull/2971
NIFI-5557: handling expired ticket by rollback and penalization
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ekovacs/nifi NIFI-5557
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2971.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2971
commit 32c42bdd971937cb6904939452420c8754f64287
Author: Endre Zoltan Kovacs
Date: 2018-08-28T08:47:59Z
NIFI-5557: handling expired ticket by rollback and penalization
> PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
> -
>
> Key: NIFI-5557
> URL: https://issues.apache.org/jira/browse/NIFI-5557
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Endre Kovacs
>Assignee: Endre Kovacs
>Priority: Major
>
> when using *PutHDFS* processor in a kerberized environment, with a flow
> "traffic" which approximately matches or less frequent then the lifetime of
> the ticket of the principal, we see this in the log:
> {code:java}
> INFO [Timer-Driven Process Thread-4] o.a.h.io.retry.RetryInvocationHandler
> Exception while invoking getFileInfo of class
> ClientNamenodeProtocolTranslatorPB over host2/ip2:8020 after 13 fail over
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for princi...@example.com to host2.example.com/ip2:8020;
> Host Details : local host is: "host1.example.com/ip1"; destination host is:
> "host2.example.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> at sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy135.getFileInfo(Unknown Source)
> at
[GitHub] nifi pull request #2971: NIFI-5557: handling expired ticket by rollback and ...
GitHub user ekovacs opened a pull request: https://github.com/apache/nifi/pull/2971 NIFI-5557: handling expired ticket by rollback and penalization Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ekovacs/nifi NIFI-5557 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2971.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2971 commit 32c42bdd971937cb6904939452420c8754f64287 Author: Endre Zoltan Kovacs Date: 2018-08-28T08:47:59Z NIFI-5557: handling expired ticket by rollback and penalization ---
[jira] [Commented] (NIFI-5542) Add support for node groups to FileAccessPolicyProvider
[ https://issues.apache.org/jira/browse/NIFI-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595137#comment-16595137 ] ASF GitHub Bot commented on NIFI-5542: -- Github user pepov commented on the issue: https://github.com/apache/nifi/pull/2970 +1 looks good to me now > Add support for node groups to FileAccessPolicyProvider > --- > > Key: NIFI-5542 > URL: https://issues.apache.org/jira/browse/NIFI-5542 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Andrew Christianson >Priority: Major > > Currently in FileAccessPolicyProvider, it is possible to specify a set of > node identities, which are given access to /proxy. This works well for static > clusters, but does not work so well for dynamic clusters (scaling up/down # > of nodes) because we don't know in advance what the node identities will be > or how many there will be. > In order to support dynamic sets of node identities, add support for > specifying a "Node Group," for which all identities in the group will be > granted access to /proxy. A UserGroupProvider can then be implemented to > gather node identities dynamically from the cluster environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2970: NIFI-5542 Added support for node groups to FileAccessPolic...
Github user pepov commented on the issue: https://github.com/apache/nifi/pull/2970 +1 looks good to me now ---
[jira] [Commented] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595129#comment-16595129
]
ASF GitHub Bot commented on NIFI-5495:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2969#discussion_r213357727
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -173,6 +175,29 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
--- End diff --
Also, some of the IT tests fail because of pretty printing causes
comparison failures.
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Assignee: Mike Thomsen
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595128#comment-16595128
]
ASF GitHub Bot commented on NIFI-5495:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2969#discussion_r213357507
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -173,6 +175,29 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
--- End diff --
Actually, this get's into the weeds if you are going to keep it required.
Here is another idea - Can we rename the Display Name of the property to
"Date Format For Standard JSON" ?
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Assignee: Mike Thomsen
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2969: NIFI-5495 Made date format configurable.
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2969#discussion_r213357727 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java --- @@ -173,6 +175,29 @@ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .build(); --- End diff -- Also, some of the IT tests fail because of pretty printing causes comparison failures. ---
[GitHub] nifi pull request #2969: NIFI-5495 Made date format configurable.
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2969#discussion_r213357507 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java --- @@ -173,6 +175,29 @@ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .build(); --- End diff -- Actually, this get's into the weeds if you are going to keep it required. Here is another idea - Can we rename the Display Name of the property to "Date Format For Standard JSON" ? ---
[jira] [Commented] (NIFI-5542) Add support for node groups to FileAccessPolicyProvider
[ https://issues.apache.org/jira/browse/NIFI-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595051#comment-16595051 ] ASF GitHub Bot commented on NIFI-5542: -- Github user achristianson commented on the issue: https://github.com/apache/nifi/pull/2970 @pepov @mcgilman new PR where we authorize the node group rather than its users. Please take a look when you get a chance. > Add support for node groups to FileAccessPolicyProvider > --- > > Key: NIFI-5542 > URL: https://issues.apache.org/jira/browse/NIFI-5542 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Andrew Christianson >Priority: Major > > Currently in FileAccessPolicyProvider, it is possible to specify a set of > node identities, which are given access to /proxy. This works well for static > clusters, but does not work so well for dynamic clusters (scaling up/down # > of nodes) because we don't know in advance what the node identities will be > or how many there will be. > In order to support dynamic sets of node identities, add support for > specifying a "Node Group," for which all identities in the group will be > granted access to /proxy. A UserGroupProvider can then be implemented to > gather node identities dynamically from the cluster environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2970: NIFI-5542 Added support for node groups to FileAccessPolic...
Github user achristianson commented on the issue: https://github.com/apache/nifi/pull/2970 @pepov @mcgilman new PR where we authorize the node group rather than its users. Please take a look when you get a chance. ---
[jira] [Commented] (NIFI-5542) Add support for node groups to FileAccessPolicyProvider
[ https://issues.apache.org/jira/browse/NIFI-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595049#comment-16595049 ] ASF GitHub Bot commented on NIFI-5542: -- GitHub user achristianson opened a pull request: https://github.com/apache/nifi/pull/2970 NIFI-5542 Added support for node groups to FileAccessPolicyProvider Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/achristianson/nifi NIFI-5542 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2970.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2970 commit 94e05575fec715a2c76dc1397e7f7124dcd1ab20 Author: Andrew I. Christianson Date: 2018-08-23T15:50:54Z NIFI-5542 Added support for node groups to FileAccessPolicyProvider > Add support for node groups to FileAccessPolicyProvider > --- > > Key: NIFI-5542 > URL: https://issues.apache.org/jira/browse/NIFI-5542 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Andrew Christianson >Priority: Major > > Currently in FileAccessPolicyProvider, it is possible to specify a set of > node identities, which are given access to /proxy. This works well for static > clusters, but does not work so well for dynamic clusters (scaling up/down # > of nodes) because we don't know in advance what the node identities will be > or how many there will be. > In order to support dynamic sets of node identities, add support for > specifying a "Node Group," for which all identities in the group will be > granted access to /proxy. A UserGroupProvider can then be implemented to > gather node identities dynamically from the cluster environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5542) Add support for node groups to FileAccessPolicyProvider
[ https://issues.apache.org/jira/browse/NIFI-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595048#comment-16595048 ] ASF GitHub Bot commented on NIFI-5542: -- Github user achristianson closed the pull request at: https://github.com/apache/nifi/pull/2965 > Add support for node groups to FileAccessPolicyProvider > --- > > Key: NIFI-5542 > URL: https://issues.apache.org/jira/browse/NIFI-5542 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Andrew Christianson >Priority: Major > > Currently in FileAccessPolicyProvider, it is possible to specify a set of > node identities, which are given access to /proxy. This works well for static > clusters, but does not work so well for dynamic clusters (scaling up/down # > of nodes) because we don't know in advance what the node identities will be > or how many there will be. > In order to support dynamic sets of node identities, add support for > specifying a "Node Group," for which all identities in the group will be > granted access to /proxy. A UserGroupProvider can then be implemented to > gather node identities dynamically from the cluster environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2970: NIFI-5542 Added support for node groups to FileAcce...
GitHub user achristianson opened a pull request: https://github.com/apache/nifi/pull/2970 NIFI-5542 Added support for node groups to FileAccessPolicyProvider Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/achristianson/nifi NIFI-5542 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2970.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2970 commit 94e05575fec715a2c76dc1397e7f7124dcd1ab20 Author: Andrew I. Christianson Date: 2018-08-23T15:50:54Z NIFI-5542 Added support for node groups to FileAccessPolicyProvider ---
[GitHub] nifi pull request #2965: NIFI-5542 Added support for node groups to FileAcce...
Github user achristianson closed the pull request at: https://github.com/apache/nifi/pull/2965 ---
[jira] [Assigned] (NIFI-5557) PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
[
https://issues.apache.org/jira/browse/NIFI-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Endre Kovacs reassigned NIFI-5557:
--
Assignee: Endre Kovacs
> PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
> -
>
> Key: NIFI-5557
> URL: https://issues.apache.org/jira/browse/NIFI-5557
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Endre Kovacs
>Assignee: Endre Kovacs
>Priority: Major
>
> when using *PutHDFS* processor in a kerberized environment, with a flow
> "traffic" which approximately matches or less frequent then the lifetime of
> the ticket of the principal, we see this in the log:
> {code:java}
> INFO [Timer-Driven Process Thread-4] o.a.h.io.retry.RetryInvocationHandler
> Exception while invoking getFileInfo of class
> ClientNamenodeProtocolTranslatorPB over host2/ip2:8020 after 13 fail over
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for princi...@example.com to host2.example.com/ip2:8020;
> Host Details : local host is: "host1.example.com/ip1"; destination host is:
> "host2.example.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> at sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy135.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:254)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
> at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:222)
> {code}
> and the flowfile is routed to failure relationship.
> *To reproduce:*
> Create a principal in your KDC with two minutes ticket lifetime,
> and set up a similar flow:
> {code:java}
> GetFile => putHDFS - success- -> logAttributes
> \
> fail
>\
> -> logAttributes
> {code}
> copy a file to the input directory of the getFile processor. If the influx
> of the flowfile is much more frequent, then the expiration time of the ticket:
> {code:java}
> watch -n 5 "cp book.txt /path/to/input"
> {code}
> then the flow will successfully run without issue.
> If we adjust this, to:
> {code:java}
> watch -n 121 "cp book.txt /path/to/input"
> {code}
> then we will observe this issue.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4371) Add support for query timeout in Hive processors
[
https://issues.apache.org/jira/browse/NIFI-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595034#comment-16595034
]
ASF GitHub Bot commented on NIFI-4371:
--
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r21538
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHive3QLProcessor.java
---
@@ -345,4 +348,22 @@ private void findTableNames(final Object obj, final
Set tableNames) {
}
return attributes;
}
+
+/**
+ * Method to set the configured timeout on the statement to be executed
+ * @param stmt statement to be executed
+ * @param context process context to retrieve the configured value
+ * @param flowFile flow file to evaluate expression language
+ * @throws ProcessException exception in case configured value cannot
be converted to an integer
+ */
+protected void setTimeout(Statement stmt, ProcessContext context,
FlowFile flowFile) throws ProcessException {
--- End diff --
The try/catch is still valid in case of error during the EL evaluation
(when EL is used to set the timeout value). I could remove the setTimeout in
the abstract class and just use setQueryTimeout directly in the Select/Put
classes but I'd still have to define how I deal with SQLException and
NumberFormatException. Honestly I think it's better to keep it as-is to have
some sort of consistency between the two bundles.
> Add support for query timeout in Hive processors
>
>
> Key: NIFI-4371
> URL: https://issues.apache.org/jira/browse/NIFI-4371
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Attachments: Screen Shot 2017-09-09 at 4.31.21 PM.png, Screen Shot
> 2017-09-09 at 6.38.51 PM.png, Screen Shot 2017-09-09 at 6.40.48 PM.png
>
>
> With HIVE-4924 it is possible to set a query timeout when executing a query
> against Hive (starting with Hive 2.1). Right now, NiFi is built using Hive
> 1.2.1 and this feature is not available by default (the method is not
> implemented in the driver). However, if building NiFi with specific profiles
> this feature can be used.
> The objective is to expose the query timeout parameter in the processor and
> enable expression language. If the version of the driver is not implementing
> the query timeout the processor will be in invalid state (unless expression
> language is used, and in this case, the flow file will be routed to the
> failure relationship).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2138: NIFI-4371 - add support for query timeout in Hive p...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r21538
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHive3QLProcessor.java
---
@@ -345,4 +348,22 @@ private void findTableNames(final Object obj, final
Set tableNames) {
}
return attributes;
}
+
+/**
+ * Method to set the configured timeout on the statement to be executed
+ * @param stmt statement to be executed
+ * @param context process context to retrieve the configured value
+ * @param flowFile flow file to evaluate expression language
+ * @throws ProcessException exception in case configured value cannot
be converted to an integer
+ */
+protected void setTimeout(Statement stmt, ProcessContext context,
FlowFile flowFile) throws ProcessException {
--- End diff --
The try/catch is still valid in case of error during the EL evaluation
(when EL is used to set the timeout value). I could remove the setTimeout in
the abstract class and just use setQueryTimeout directly in the Select/Put
classes but I'd still have to define how I deal with SQLException and
NumberFormatException. Honestly I think it's better to keep it as-is to have
some sort of consistency between the two bundles.
---
[jira] [Commented] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595007#comment-16595007
]
ASF GitHub Bot commented on NIFI-5495:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2969#discussion_r213326155
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -173,6 +175,29 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
--- End diff --
Sure.
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Assignee: Mike Thomsen
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2969: NIFI-5495 Made date format configurable.
Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/2969#discussion_r213326155 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java --- @@ -173,6 +175,29 @@ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .build(); --- End diff -- Sure. ---
[jira] [Commented] (NIFI-4371) Add support for query timeout in Hive processors
[
https://issues.apache.org/jira/browse/NIFI-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594988#comment-16594988
]
ASF GitHub Bot commented on NIFI-4371:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r213318454
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHive3QLProcessor.java
---
@@ -345,4 +348,22 @@ private void findTableNames(final Object obj, final
Set tableNames) {
}
return attributes;
}
+
+/**
+ * Method to set the configured timeout on the statement to be executed
+ * @param stmt statement to be executed
+ * @param context process context to retrieve the configured value
+ * @param flowFile flow file to evaluate expression language
+ * @throws ProcessException exception in case configured value cannot
be converted to an integer
+ */
+protected void setTimeout(Statement stmt, ProcessContext context,
FlowFile flowFile) throws ProcessException {
--- End diff --
I mentioned it only for consistency between the two sets of processors,
actually I'd be fine with not even putting the setQueryTimeout in a try/catch
for the Hive 3 processors, now that it's supported, I don't imagine there are
any drivers that wouldn't support it at this point.
Either way you want to go is fine with me, let me know what you end up with
(even if it is what you already have :) and I'll finish the review and merge,
thanks!
> Add support for query timeout in Hive processors
>
>
> Key: NIFI-4371
> URL: https://issues.apache.org/jira/browse/NIFI-4371
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Attachments: Screen Shot 2017-09-09 at 4.31.21 PM.png, Screen Shot
> 2017-09-09 at 6.38.51 PM.png, Screen Shot 2017-09-09 at 6.40.48 PM.png
>
>
> With HIVE-4924 it is possible to set a query timeout when executing a query
> against Hive (starting with Hive 2.1). Right now, NiFi is built using Hive
> 1.2.1 and this feature is not available by default (the method is not
> implemented in the driver). However, if building NiFi with specific profiles
> this feature can be used.
> The objective is to expose the query timeout parameter in the processor and
> enable expression language. If the version of the driver is not implementing
> the query timeout the processor will be in invalid state (unless expression
> language is used, and in this case, the flow file will be routed to the
> failure relationship).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2138: NIFI-4371 - add support for query timeout in Hive p...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r213318454
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHive3QLProcessor.java
---
@@ -345,4 +348,22 @@ private void findTableNames(final Object obj, final
Set tableNames) {
}
return attributes;
}
+
+/**
+ * Method to set the configured timeout on the statement to be executed
+ * @param stmt statement to be executed
+ * @param context process context to retrieve the configured value
+ * @param flowFile flow file to evaluate expression language
+ * @throws ProcessException exception in case configured value cannot
be converted to an integer
+ */
+protected void setTimeout(Statement stmt, ProcessContext context,
FlowFile flowFile) throws ProcessException {
--- End diff --
I mentioned it only for consistency between the two sets of processors,
actually I'd be fine with not even putting the setQueryTimeout in a try/catch
for the Hive 3 processors, now that it's supported, I don't imagine there are
any drivers that wouldn't support it at this point.
Either way you want to go is fine with me, let me know what you end up with
(even if it is what you already have :) and I'll finish the review and merge,
thanks!
---
[GitHub] nifi pull request #2969: NIFI-5495 Made date format configurable.
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2969#discussion_r213317913 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java --- @@ -173,6 +175,29 @@ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .build(); --- End diff -- would you accept a PR on your ... PR? ---
[jira] [Commented] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594985#comment-16594985
]
ASF GitHub Bot commented on NIFI-5495:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2969#discussion_r213317913
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -173,6 +175,29 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
--- End diff --
would you accept a PR on your ... PR?
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Assignee: Mike Thomsen
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594970#comment-16594970
]
ASF GitHub Bot commented on NIFI-5495:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2969#discussion_r213312718
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -173,6 +175,29 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
--- End diff --
Right, the code will work. The user however may be a bit surprised. "Why
let me configure it, and then ignore what I put in without telling me?"
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Assignee: Mike Thomsen
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2969: NIFI-5495 Made date format configurable.
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2969#discussion_r213312718 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java --- @@ -173,6 +175,29 @@ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .build(); --- End diff -- Right, the code will work. The user however may be a bit surprised. "Why let me configure it, and then ignore what I put in without telling me?" ---
[jira] [Commented] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594948#comment-16594948
]
ASF GitHub Bot commented on NIFI-5495:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2969#discussion_r213305700
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -173,6 +175,29 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
--- End diff --
If they have them both set, it's not a problem. The processor just ignores
the date format configuration.
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Assignee: Mike Thomsen
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2969: NIFI-5495 Made date format configurable.
Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/2969#discussion_r213305700 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java --- @@ -173,6 +175,29 @@ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .build(); --- End diff -- If they have them both set, it's not a problem. The processor just ignores the date format configuration. ---
[jira] [Comment Edited] (NIFI-5151) Patch Nifi with Upsert functions for PutDatabaseRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594864#comment-16594864 ] Karl Amundsson edited comment on NIFI-5151 at 8/28/18 11:47 AM: Thank you for your detailed comment and sorry for my late answer! I wanted to patch it before I responded, and then I got busy with other stuff. I've now patched it to use DatabaseAdapters. was (Author: kamund): Thank you for your detailed comment and sorry for my late answer! I wanted to patch it before I responded, and then I got busy with other stuff. I've now patched it to use DatabaseAdapters, let me know if this is what you had in mind. > Patch Nifi with Upsert functions for PutDatabaseRecord processor > > > Key: NIFI-5151 > URL: https://issues.apache.org/jira/browse/NIFI-5151 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.0 >Reporter: Karl Amundsson >Priority: Major > Labels: Processor > Attachments: > 0001-NIFI-5151-Adding-support-for-UPSERT-in-PutDatabaseRe.patch, > 0001-NIFI-5151-Using-DatabaseAdapter-to-generate-INSERT-S.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > Since Phoenix doesn't support the SQL statement INSERT you have to use a > process like: ConvertAttributesToJSON->ConvertJSONToSQL in Insert > mode->ReplaceText to replace "INSERT" with "UPSERT" -> PutSQL (See: > [https://community.hortonworks.com/questions/40561/nifi-phoenix-processor.html)] > With this patch you can choose to use UPSERT directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5151) Patch Nifi with Upsert functions for PutDatabaseRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594864#comment-16594864 ] Karl Amundsson commented on NIFI-5151: -- Thank you for your detailed comment and sorry for my late answer! I wanted to patch it before I responded, and then I got busy with other stuff. I've now patched it to use DatabaseAdapters, let me know if this is what you had in mind. > Patch Nifi with Upsert functions for PutDatabaseRecord processor > > > Key: NIFI-5151 > URL: https://issues.apache.org/jira/browse/NIFI-5151 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.0 >Reporter: Karl Amundsson >Priority: Major > Labels: Processor > Attachments: > 0001-NIFI-5151-Adding-support-for-UPSERT-in-PutDatabaseRe.patch, > 0001-NIFI-5151-Using-DatabaseAdapter-to-generate-INSERT-S.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > Since Phoenix doesn't support the SQL statement INSERT you have to use a > process like: ConvertAttributesToJSON->ConvertJSONToSQL in Insert > mode->ReplaceText to replace "INSERT" with "UPSERT" -> PutSQL (See: > [https://community.hortonworks.com/questions/40561/nifi-phoenix-processor.html)] > With this patch you can choose to use UPSERT directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5151) Patch Nifi with Upsert functions for PutDatabaseRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Amundsson updated NIFI-5151: - Attachment: 0001-NIFI-5151-Using-DatabaseAdapter-to-generate-INSERT-S.patch > Patch Nifi with Upsert functions for PutDatabaseRecord processor > > > Key: NIFI-5151 > URL: https://issues.apache.org/jira/browse/NIFI-5151 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.0 >Reporter: Karl Amundsson >Priority: Major > Labels: Processor > Attachments: > 0001-NIFI-5151-Adding-support-for-UPSERT-in-PutDatabaseRe.patch, > 0001-NIFI-5151-Using-DatabaseAdapter-to-generate-INSERT-S.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > Since Phoenix doesn't support the SQL statement INSERT you have to use a > process like: ConvertAttributesToJSON->ConvertJSONToSQL in Insert > mode->ReplaceText to replace "INSERT" with "UPSERT" -> PutSQL (See: > [https://community.hortonworks.com/questions/40561/nifi-phoenix-processor.html)] > With this patch you can choose to use UPSERT directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5557) PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
[
https://issues.apache.org/jira/browse/NIFI-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594742#comment-16594742
]
Endre Kovacs commented on NIFI-5557:
Please assign me this issue, as I already have a proposed fix.
Thanks!
Endre
> PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
> -
>
> Key: NIFI-5557
> URL: https://issues.apache.org/jira/browse/NIFI-5557
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Endre Kovacs
>Priority: Major
>
> when using *PutHDFS* processor in a kerberized environment, with a flow
> "traffic" which approximately matches or less frequent then the lifetime of
> the ticket of the principal, we see this in the log:
> {code:java}
> INFO [Timer-Driven Process Thread-4] o.a.h.io.retry.RetryInvocationHandler
> Exception while invoking getFileInfo of class
> ClientNamenodeProtocolTranslatorPB over host2/ip2:8020 after 13 fail over
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for princi...@example.com to host2.example.com/ip2:8020;
> Host Details : local host is: "host1.example.com/ip1"; destination host is:
> "host2.example.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> at sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy135.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:254)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
> at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:222)
> {code}
> and the flowfile is routed to failure relationship.
> *To reproduce:*
> Create a principal in your KDC with two minutes ticket lifetime,
> and set up a similar flow:
> {code:java}
> GetFile => putHDFS - success- -> logAttributes
> \
> fail
>\
> -> logAttributes
> {code}
> copy a file to the input directory of the getFile processor. If the influx
> of the flowfile is much more frequent, then the expiration time of the ticket:
> {code:java}
> watch -n 5 "cp book.txt /path/to/input"
> {code}
> then the flow will successfully run without issue.
> If we adjust this, to:
> {code:java}
> watch -n 121 "cp book.txt /path/to/input"
> {code}
> then we will observe this issue.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5557) PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
Endre Kovacs created NIFI-5557:
--
Summary: PutHDFS "GSSException: No valid credentials provided"
when krb ticket expires
Key: NIFI-5557
URL: https://issues.apache.org/jira/browse/NIFI-5557
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.5.0
Reporter: Endre Kovacs
when using *PutHDFS* processor in a kerberized environment, with a flow
"traffic" which approximately matches or less frequent then the lifetime of the
ticket of the principal, we see this in the log:
{code:java}
INFO [Timer-Driven Process Thread-4] o.a.h.io.retry.RetryInvocationHandler
Exception while invoking getFileInfo of class
ClientNamenodeProtocolTranslatorPB over host2/ip2:8020 after 13 fail over
attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
setup connection for princi...@example.com to host2.example.com/ip2:8020; Host
Details : local host is: "host1.example.com/ip1"; destination host is:
"host2.example.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy135.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:254)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:222)
{code}
and the flowfile is routed to failure relationship.
*To reproduce:*
Create a principal in your KDC with two minutes ticket lifetime,
and set up a similar flow:
{code:java}
GetFile => putHDFS - success- -> logAttributes
\
fail
\
-> logAttributes
{code}
copy a file to the input directory of the getFile processor. If the influx of
the flowfile is much more frequent, then the expiration time of the ticket:
{code:java}
watch -n 5 "cp book.txt /path/to/input"
{code}
then the flow will successfully run without issue.
If we adjust this, to:
{code:java}
watch -n 121 "cp book.txt /path/to/input"
{code}
then we will observe this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFIREG-197) Flow and Bucket is not retrieved when cloned from from Git to NiFi Registry
Varshinee created NIFIREG-197: - Summary: Flow and Bucket is not retrieved when cloned from from Git to NiFi Registry Key: NIFIREG-197 URL: https://issues.apache.org/jira/browse/NIFIREG-197 Project: NiFi Registry Issue Type: Bug Affects Versions: 0.2.0 Reporter: Varshinee What I did - # I have two NiFi Registry for Dev and Prod. I have pushed the flow from Dev NiFi Registry to Git by using GitFlowPersistenceProvider. # Tried to retrieve the flow and bucket from Git for the Prod machine by git clone. # When checking Prod NiFi Registry the flows are not available. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4371) Add support for query timeout in Hive processors
[
https://issues.apache.org/jira/browse/NIFI-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594631#comment-16594631
]
ASF GitHub Bot commented on NIFI-4371:
--
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r213204485
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHive3QLProcessor.java
---
@@ -345,4 +348,22 @@ private void findTableNames(final Object obj, final
Set tableNames) {
}
return attributes;
}
+
+/**
+ * Method to set the configured timeout on the statement to be executed
+ * @param stmt statement to be executed
+ * @param context process context to retrieve the configured value
+ * @param flowFile flow file to evaluate expression language
+ * @throws ProcessException exception in case configured value cannot
be converted to an integer
+ */
+protected void setTimeout(Statement stmt, ProcessContext context,
FlowFile flowFile) throws ProcessException {
--- End diff --
Yeah... decided not to because I figured that all the drivers on Hive3 and
beyond would implement the method now. I thought it would make less validation
work when the framework is validating the processor. But I guess we could
choose to be on the safest side and add it anyway. Thoughts?
> Add support for query timeout in Hive processors
>
>
> Key: NIFI-4371
> URL: https://issues.apache.org/jira/browse/NIFI-4371
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Attachments: Screen Shot 2017-09-09 at 4.31.21 PM.png, Screen Shot
> 2017-09-09 at 6.38.51 PM.png, Screen Shot 2017-09-09 at 6.40.48 PM.png
>
>
> With HIVE-4924 it is possible to set a query timeout when executing a query
> against Hive (starting with Hive 2.1). Right now, NiFi is built using Hive
> 1.2.1 and this feature is not available by default (the method is not
> implemented in the driver). However, if building NiFi with specific profiles
> this feature can be used.
> The objective is to expose the query timeout parameter in the processor and
> enable expression language. If the version of the driver is not implementing
> the query timeout the processor will be in invalid state (unless expression
> language is used, and in this case, the flow file will be routed to the
> failure relationship).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2138: NIFI-4371 - add support for query timeout in Hive p...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r213204485
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHive3QLProcessor.java
---
@@ -345,4 +348,22 @@ private void findTableNames(final Object obj, final
Set tableNames) {
}
return attributes;
}
+
+/**
+ * Method to set the configured timeout on the statement to be executed
+ * @param stmt statement to be executed
+ * @param context process context to retrieve the configured value
+ * @param flowFile flow file to evaluate expression language
+ * @throws ProcessException exception in case configured value cannot
be converted to an integer
+ */
+protected void setTimeout(Statement stmt, ProcessContext context,
FlowFile flowFile) throws ProcessException {
--- End diff --
Yeah... decided not to because I figured that all the drivers on Hive3 and
beyond would implement the method now. I thought it would make less validation
work when the framework is validating the processor. But I guess we could
choose to be on the safest side and add it anyway. Thoughts?
---
[jira] [Updated] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5479: - Resolution: Fixed Fix Version/s: 1.8.0 Status: Resolved (was: Patch Available) > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Major > Fix For: 1.8.0 > > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594589#comment-16594589 ] ASF subversion and git services commented on NIFI-5479: --- Commit f04cd8681d3532f9c8f2eb7306962056a287e2ad in nifi's branch refs/heads/master from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=f04cd86 ] NIFI-5479: Using the SUN provider when the keystore type is JKS. This closes #2961. Signed-off-by: Koji Kawamura > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Major > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594588#comment-16594588 ] ASF subversion and git services commented on NIFI-5479: --- Commit a27ccd8a56fb52a61d79f87711324142a5b4a141 in nifi's branch refs/heads/master from joewitt [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a27ccd8 ] NIFI-5479 Upgraded Jetty. Moved where we unpack bundled deps to so we can avoid a new jetty bug with META-INF loading logic. Signed-off-by: Koji Kawamura > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Major > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594590#comment-16594590 ] ASF GitHub Bot commented on NIFI-5479: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2961 > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Major > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2961: NIFI-5479: Upgrade Jetty
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2961 ---
[jira] [Commented] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594580#comment-16594580 ] ASF GitHub Bot commented on NIFI-5479: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2961 @mcgilman I've confirmed that NiFi Websocket component uses the updated Jetty version and works without issue. I'm a +1 on this, too. Thanks for this fix, merging! > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Major > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2961: NIFI-5479: Upgrade Jetty
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2961 @mcgilman I've confirmed that NiFi Websocket component uses the updated Jetty version and works without issue. I'm a +1 on this, too. Thanks for this fix, merging! ---
