[jira] [Reopened] (NIFI-4558) Populate default keystore/truststore types in SSLContextService
[
https://issues.apache.org/jira/browse/NIFI-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough reopened NIFI-4558:
Found that this does not work as expected. The default value is not passed to
the customValidate() method, and as such it receives a null for the
keystore/truststore type.
> Populate default keystore/truststore types in SSLContextService
> ---
>
> Key: NIFI-4558
> URL: https://issues.apache.org/jira/browse/NIFI-4558
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Trivial
> Labels: controller_services, jks, keystore, pkcs12, security,
> ssl, tls, truststore, ux
> Fix For: 1.8.0
>
>
> The keystore and truststore type is almost always JKS as opposed to PKCS12
> when creating SSL controller services. Both {{StandardSSLContextService}} and
> {{StandardRestrictedSSLContextService}} should have those fields
> autopopulated to JKS, saving 2-4 clicks per instantiation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4558) Populate default keystore/truststore types in SSLContextService
[
https://issues.apache.org/jira/browse/NIFI-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657379#comment-16657379
]
ASF GitHub Bot commented on NIFI-4558:
--
GitHub user thenatog opened a pull request:
https://github.com/apache/nifi/pull/3097
Revert "NIFI-4558 - Set JKS as the default keystore type and truststo…
…re type."
This reverts commit 2b9fdf8415eb2b033bfc63a6c5254bba3e69ecdd.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/thenatog/nifi NIFI-4558-rebased
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3097.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3097
commit ddc4e28967c55ebe3522c16dc1e0bd642936e534
Author: thenatog
Date: 2018-10-19T19:46:12Z
Revert "NIFI-4558 - Set JKS as the default keystore type and truststore
type."
This reverts commit 2b9fdf8415eb2b033bfc63a6c5254bba3e69ecdd.
> Populate default keystore/truststore types in SSLContextService
> ---
>
> Key: NIFI-4558
> URL: https://issues.apache.org/jira/browse/NIFI-4558
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Trivial
> Labels: controller_services, jks, keystore, pkcs12, security,
> ssl, tls, truststore, ux
> Fix For: 1.8.0
>
>
> The keystore and truststore type is almost always JKS as opposed to PKCS12
> when creating SSL controller services. Both {{StandardSSLContextService}} and
> {{StandardRestrictedSSLContextService}} should have those fields
> autopopulated to JKS, saving 2-4 clicks per instantiation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3097: Revert "NIFI-4558 - Set JKS as the default keystore...
GitHub user thenatog opened a pull request: https://github.com/apache/nifi/pull/3097 Revert "NIFI-4558 - Set JKS as the default keystore type and truststo⦠â¦re type." This reverts commit 2b9fdf8415eb2b033bfc63a6c5254bba3e69ecdd. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/thenatog/nifi NIFI-4558-rebased Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3097.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3097 commit ddc4e28967c55ebe3522c16dc1e0bd642936e534 Author: thenatog Date: 2018-10-19T19:46:12Z Revert "NIFI-4558 - Set JKS as the default keystore type and truststore type." This reverts commit 2b9fdf8415eb2b033bfc63a6c5254bba3e69ecdd. ---
[jira] [Comment Edited] (NIFI-5102) MarkLogic DB Processors
[ https://issues.apache.org/jira/browse/NIFI-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657377#comment-16657377 ] Yash Tambawala edited comment on NIFI-5102 at 10/19/18 8:25 PM: Hello Joe. We've been working on removing a couple of bugs for this processor and have updated the PR. We have also created a step-by-step guide for trying out our processors - [https://marklogic.github.io/nifi/step-by-step] from a function perspective, as you had requested. was (Author: yash.tambawala): Hello Joe. We've been working on removing a couple of bugs for this processor and have updated the PR. We have also created a step-by-step guide for trying out our processors - [https://marklogic.github.io/nifi/step-by-step] from a function perspective, as you had requested. > MarkLogic DB Processors > --- > > Key: NIFI-5102 > URL: https://issues.apache.org/jira/browse/NIFI-5102 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Anthony Roach >Priority: Major > Attachments: > 0002-NIFI-5102-making-some-updates-to-help-but-security-n.patch > > > As a data architect, I need to ingest data from my NiFi FlowFile into > MarkLogic database documents. I have created the following two processors: > * PutMarkLogic: Ingest Flowfile into MarkLogic database documents > * QueryMarkLogic: Retrieve result set from MarkLogic into FlowFile > I will create a pull request. > [www.marklogic.com|http://www.marklogic.com/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5102) MarkLogic DB Processors
[ https://issues.apache.org/jira/browse/NIFI-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657377#comment-16657377 ] Yash Tambawala commented on NIFI-5102: -- Hello Joe. We've been working on removing a couple of bugs for this processor and have updated the PR. We have also created a step-by-step guide for trying out our processors - [https://marklogic.github.io/nifi/step-by-step] from a function perspective, as you had requested. > MarkLogic DB Processors > --- > > Key: NIFI-5102 > URL: https://issues.apache.org/jira/browse/NIFI-5102 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Anthony Roach >Priority: Major > Attachments: > 0002-NIFI-5102-making-some-updates-to-help-but-security-n.patch > > > As a data architect, I need to ingest data from my NiFi FlowFile into > MarkLogic database documents. I have created the following two processors: > * PutMarkLogic: Ingest Flowfile into MarkLogic database documents > * QueryMarkLogic: Retrieve result set from MarkLogic into FlowFile > I will create a pull request. > [www.marklogic.com|http://www.marklogic.com/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-623) Add thread backtrace to describe command
[ https://issues.apache.org/jira/browse/MINIFICPP-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657194#comment-16657194 ] ASF GitHub Bot commented on MINIFICPP-623: -- Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/424 great, thanks! > Add thread backtrace to describe command > > > Key: MINIFICPP-623 > URL: https://issues.apache.org/jira/browse/MINIFICPP-623 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Mr TheSegfault >Priority: Major > Labels: c2, devops > > We should be able perform a remote jstack ( as it were ) to get debug > information remotely through the predefined c2 capabilities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #424: MINIFICPP-623: Add trace capabilities to control...
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/424 great, thanks! ---
[jira] [Commented] (MINIFICPP-623) Add thread backtrace to describe command
[ https://issues.apache.org/jira/browse/MINIFICPP-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657180#comment-16657180 ] ASF GitHub Bot commented on MINIFICPP-623: -- Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/424 @apiri yeah it seems that it's because they are run in parallel and there is a port collision. I am changing port numbers in the tests. > Add thread backtrace to describe command > > > Key: MINIFICPP-623 > URL: https://issues.apache.org/jira/browse/MINIFICPP-623 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Mr TheSegfault >Priority: Major > Labels: c2, devops > > We should be able perform a remote jstack ( as it were ) to get debug > information remotely through the predefined c2 capabilities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #424: MINIFICPP-623: Add trace capabilities to control...
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/424 @apiri yeah it seems that it's because they are run in parallel and there is a port collision. I am changing port numbers in the tests. ---
[jira] [Commented] (MINIFICPP-623) Add thread backtrace to describe command
[ https://issues.apache.org/jira/browse/MINIFICPP-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657174#comment-16657174 ] ASF GitHub Bot commented on MINIFICPP-623: -- Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/424 Haven't really dug in, but I did see that Travis is reporting that the C2Jstack test is failing. Seems like it might be environmental for Travis but haven't inspected too much. > Add thread backtrace to describe command > > > Key: MINIFICPP-623 > URL: https://issues.apache.org/jira/browse/MINIFICPP-623 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Mr TheSegfault >Priority: Major > Labels: c2, devops > > We should be able perform a remote jstack ( as it were ) to get debug > information remotely through the predefined c2 capabilities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #424: MINIFICPP-623: Add trace capabilities to control...
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/424 Haven't really dug in, but I did see that Travis is reporting that the C2Jstack test is failing. Seems like it might be environmental for Travis but haven't inspected too much. ---
[GitHub] nifi-minifi-cpp issue #424: MINIFICPP-623: Add trace capabilities to control...
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/424 reviewing ---
[jira] [Commented] (MINIFICPP-623) Add thread backtrace to describe command
[ https://issues.apache.org/jira/browse/MINIFICPP-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657169#comment-16657169 ] ASF GitHub Bot commented on MINIFICPP-623: -- Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/424 reviewing > Add thread backtrace to describe command > > > Key: MINIFICPP-623 > URL: https://issues.apache.org/jira/browse/MINIFICPP-623 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Mr TheSegfault >Priority: Major > Labels: c2, devops > > We should be able perform a remote jstack ( as it were ) to get debug > information remotely through the predefined c2 capabilities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5673) Support auto loading of new NARs
[ https://issues.apache.org/jira/browse/NIFI-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657124#comment-16657124 ] ASF GitHub Bot commented on NIFI-5673: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/3095 @bbende can you change the title to refer to the correct jira, you got me all excited about the other jira ;) > Support auto loading of new NARs > > > Key: NIFI-5673 > URL: https://issues.apache.org/jira/browse/NIFI-5673 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > > We should be able to detect when new NARs have been added to any of the NAR > directories and automatically load them and make the components available for > use without restarting the whole NiFi instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3095: NIFI-5673 Fixing connection handling in MQTT processors
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/3095 @bbende can you change the title to refer to the correct jira, you got me all excited about the other jira ;) ---
[jira] [Commented] (NIFI-5721) ConsumeMQTT processor can exhaust system threads
[ https://issues.apache.org/jira/browse/NIFI-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657122#comment-16657122 ] ASF GitHub Bot commented on NIFI-5721: -- GitHub user bbende opened a pull request: https://github.com/apache/nifi/pull/3096 NIFI-5721 Fixing connection handling in MQTT processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbende/nifi NIFI-5721 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3096.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3096 commit 39bfebaf0c00f213f068f128228e3b271ad05686 Author: Bryan Bende Date: 2018-10-18T21:02:42Z NIFI-5721 Fixing connection handling in MQTT processors > ConsumeMQTT processor can exhaust system threads > > > Key: NIFI-5721 > URL: https://issues.apache.org/jira/browse/NIFI-5721 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Aldrin Piri >Assignee: Bryan Bende >Priority: Major > Attachments: ConsumeMQTT_OOME.xml, nifi-app-oome.log > > > Originally reported on the mailing list as a MiNiFi issue, > https://lists.apache.org/thread.html/d64afc392d8c503997634ed4e9285ba4e89ccc83002e99c66d259c89@%3Cusers.nifi.apache.org%3E. > After some initial evaluation it appears there is a bug being triggered in > the mqtt library that is not being handled by NiFi and causing a resource > leak. > Original thread here: > https://lists.apache.org/thread.html/d64afc392d8c503997634ed4e9285ba4e89ccc83002e99c66d259c89@%3Cusers.nifi.apache.org%3E > Additional details to follow as I evaluate how to trigger it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3096: NIFI-5721 Fixing connection handling in MQTT proces...
GitHub user bbende opened a pull request: https://github.com/apache/nifi/pull/3096 NIFI-5721 Fixing connection handling in MQTT processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbende/nifi NIFI-5721 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3096.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3096 commit 39bfebaf0c00f213f068f128228e3b271ad05686 Author: Bryan Bende Date: 2018-10-18T21:02:42Z NIFI-5721 Fixing connection handling in MQTT processors ---
[jira] [Created] (NIFI-5732) Provide configuration of load balancing functionality in Docker images
Aldrin Piri created NIFI-5732: - Summary: Provide configuration of load balancing functionality in Docker images Key: NIFI-5732 URL: https://issues.apache.org/jira/browse/NIFI-5732 Project: Apache NiFi Issue Type: Improvement Components: Docker Reporter: Aldrin Piri With the implementation of NIFI-5516, it would be helpful to provide support for this with the clustering setup in our Docker images. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5673) Support auto loading of new NARs
[ https://issues.apache.org/jira/browse/NIFI-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657120#comment-16657120 ] ASF GitHub Bot commented on NIFI-5673: -- Github user bbende closed the pull request at: https://github.com/apache/nifi/pull/3095 > Support auto loading of new NARs > > > Key: NIFI-5673 > URL: https://issues.apache.org/jira/browse/NIFI-5673 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > > We should be able to detect when new NARs have been added to any of the NAR > directories and automatically load them and make the components available for > use without restarting the whole NiFi instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3095: NIFI-5673 Fixing connection handling in MQTT proces...
Github user bbende closed the pull request at: https://github.com/apache/nifi/pull/3095 ---
[jira] [Updated] (NIFI-5721) ConsumeMQTT processor can exhaust system threads
[ https://issues.apache.org/jira/browse/NIFI-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-5721: -- Fix Version/s: (was: 1.7.1) Affects Version/s: 1.7.1 Status: Patch Available (was: Open) > ConsumeMQTT processor can exhaust system threads > > > Key: NIFI-5721 > URL: https://issues.apache.org/jira/browse/NIFI-5721 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Aldrin Piri >Assignee: Bryan Bende >Priority: Major > Attachments: ConsumeMQTT_OOME.xml, nifi-app-oome.log > > > Originally reported on the mailing list as a MiNiFi issue, > https://lists.apache.org/thread.html/d64afc392d8c503997634ed4e9285ba4e89ccc83002e99c66d259c89@%3Cusers.nifi.apache.org%3E. > After some initial evaluation it appears there is a bug being triggered in > the mqtt library that is not being handled by NiFi and causing a resource > leak. > Original thread here: > https://lists.apache.org/thread.html/d64afc392d8c503997634ed4e9285ba4e89ccc83002e99c66d259c89@%3Cusers.nifi.apache.org%3E > Additional details to follow as I evaluate how to trigger it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5673) Support auto loading of new NARs
[ https://issues.apache.org/jira/browse/NIFI-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657114#comment-16657114 ] ASF GitHub Bot commented on NIFI-5673: -- GitHub user bbende opened a pull request: https://github.com/apache/nifi/pull/3095 NIFI-5673 Fixing connection handling in MQTT processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbende/nifi NIFI-5721 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3095.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3095 commit 15a0ef46f7ab6b87b949f055ec57d4bf72541099 Author: Bryan Bende Date: 2018-10-18T21:02:42Z NIFI-5673 Fixing connection handling in MQTT processors > Support auto loading of new NARs > > > Key: NIFI-5673 > URL: https://issues.apache.org/jira/browse/NIFI-5673 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > > We should be able to detect when new NARs have been added to any of the NAR > directories and automatically load them and make the components available for > use without restarting the whole NiFi instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3095: NIFI-5673 Fixing connection handling in MQTT proces...
GitHub user bbende opened a pull request: https://github.com/apache/nifi/pull/3095 NIFI-5673 Fixing connection handling in MQTT processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbende/nifi NIFI-5721 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3095.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3095 commit 15a0ef46f7ab6b87b949f055ec57d4bf72541099 Author: Bryan Bende Date: 2018-10-18T21:02:42Z NIFI-5673 Fixing connection handling in MQTT processors ---
[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r226306211
--- Diff: libminifi/include/capi/Plan.h ---
@@ -93,11 +95,50 @@ class ExecutionPlan {
next_ff_ = ptr;
}
+ static std::shared_ptr createProcessor(const
std::string &processor_name, const std::string &name);
+
protected:
+ class FailureHandler {
+ public:
+FailureHandler() {
+ callback_ = nullptr;
+}
+void setCallback(void (*onerror_callback)(const flow_file_record*)) {
+ callback_=onerror_callback;
+}
+void operator()(const processor_session* ps)
+{
+ auto ses = static_cast(ps->session);
+
+ auto ff = ses->get();
+ if (ff == nullptr) {
+return;
+ }
+ auto claim = ff->getResourceClaim();
+
+ if (claim != nullptr && callback_ != nullptr) {
+// create a flow file.
+auto path = claim->getContentFullPath();
+auto ffr = create_ff_object_na(path.c_str(), path.length(),
ff->getSize());
+ffr->attributes = ff->getAttributesPtr();
+ffr->ffp = ff.get();
+callback_(ffr);
+ }
+ // This deletes the content of the flowfile as ff gets out of scope
+ // It's the users responsibility to copy all the data
+ ses->remove(ff);
+
+}
+ private:
+void (*callback_)(const flow_file_record*);
--- End diff --
^ no std::function here either?
---
[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r226309091
--- Diff: libminifi/src/capi/Plan.cpp ---
@@ -202,29 +180,31 @@ std::shared_ptr
ExecutionPlan::getCurrentSession() {
return current_session_;
}
-std::shared_ptr
ExecutionPlan::buildFinalConnection(std::shared_ptr processor,
bool setDest) {
- std::stringstream connection_name;
- std::shared_ptr last = processor;
- connection_name << last->getUUIDStr() << "-to-" <<
processor->getUUIDStr();
- std::shared_ptr connection =
std::make_shared(flow_repo_, content_repo_,
connection_name.str());
- connection->setRelationship(termination_);
+std::shared_ptr
ExecutionPlan::buildFinalConnection(std::shared_ptr processor,
bool set_dst) {
+ return connectProcessors(processor, processor, termination_, set_dst);
+}
- // link the connections so that we can test results at the end for this
- connection->setSource(last);
- if (setDest)
-connection->setDestination(processor);
+void ExecutionPlan::finalize() {
+ if (failure_handler_) {
+auto failure_proc = createProcessor("CallbackProcessor",
"CallbackProcessor");
- utils::Identifier uuid_copy;
- last->getUUID(uuid_copy);
- connection->setSourceUUID(uuid_copy);
- if (setDest)
-connection->setDestinationUUID(uuid_copy);
+std::shared_ptr callback_proc =
std::static_pointer_cast(failure_proc);
+callback_proc->setCallback(nullptr,
std::bind(&FailureHandler::operator(), failure_handler_,
std::placeholders::_1));
- processor->addConnection(connection);
- return connection;
-}
+for (const auto& proc : processor_queue_) {
+ relationships_.push_back(connectProcessors(proc, failure_proc,
core::Relationship("failure", "failure collector"), true));
--- End diff --
Are you checking processors to see if they have a failure relationship?
---
[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r226717192 --- Diff: libminifi/include/capi/api.h --- @@ -79,6 +79,13 @@ processor *add_processor_with_linkage(flow *flow, const char *processor_name); processor *add_python_processor(flow *, void (*ontrigger_callback)(processor_session *session)); +/** +* Register your callback to received flow files that the flow failed to process +* The flow file is deleted after the callback is executed, make sure to copy all the data you need! +* The first callback should be registered before the flow is used. Can be changed later during runtime. +*/ --- End diff -- what if failure is an optional condition? Why remove and not reuse? curious about comment above. ---
[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r226716905
--- Diff: libminifi/include/capi/Plan.h ---
@@ -93,11 +95,50 @@ class ExecutionPlan {
next_ff_ = ptr;
}
+ static std::shared_ptr createProcessor(const
std::string &processor_name, const std::string &name);
+
protected:
+ class FailureHandler {
+ public:
+FailureHandler() {
+ callback_ = nullptr;
+}
+void setCallback(void (*onerror_callback)(const flow_file_record*)) {
+ callback_=onerror_callback;
+}
+void operator()(const processor_session* ps)
+{
--- End diff --
hmm, this doesn't give the option of what's called a rollback ( go back and
potentially re-try). I'm curious if this could be done more simply with a
function ptr and that gives them the option and provide them a set of
ready-made ones that rollback and retry, drop, and continue, etc?
---
[GitHub] nifi-minifi-cpp pull request #421: Minificpp 641 - C API: add support to reg...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/421#discussion_r226302333
--- Diff: libminifi/include/capi/Plan.h ---
@@ -93,11 +95,50 @@ class ExecutionPlan {
next_ff_ = ptr;
}
+ static std::shared_ptr createProcessor(const
std::string &processor_name, const std::string &name);
+
protected:
+ class FailureHandler {
+ public:
+FailureHandler() {
+ callback_ = nullptr;
+}
+void setCallback(void (*onerror_callback)(const flow_file_record*)) {
--- End diff --
std::function for consistency?
---
[jira] [Commented] (MINIFICPP-623) Add thread backtrace to describe command
[ https://issues.apache.org/jira/browse/MINIFICPP-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656987#comment-16656987 ] ASF GitHub Bot commented on MINIFICPP-623: -- GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/424 MINIFICPP-623: Add trace capabilities to controller and agent Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-623 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/424.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #424 commit 97de1cd29a54c7d14116859aa5aeffee977a05f7 Author: Marc Parisi Date: 2018-10-19T00:13:15Z MINIFICPP-623: Add trace capabilities to controller and agent > Add thread backtrace to describe command > > > Key: MINIFICPP-623 > URL: https://issues.apache.org/jira/browse/MINIFICPP-623 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Mr TheSegfault >Priority: Major > Labels: c2, devops > > We should be able perform a remote jstack ( as it were ) to get debug > information remotely through the predefined c2 capabilities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #424: MINIFICPP-623: Add trace capabilities to ...
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/424 MINIFICPP-623: Add trace capabilities to controller and agent Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-623 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/424.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #424 commit 97de1cd29a54c7d14116859aa5aeffee977a05f7 Author: Marc Parisi Date: 2018-10-19T00:13:15Z MINIFICPP-623: Add trace capabilities to controller and agent ---
[jira] [Created] (NIFI-5731) Support List-typed fields for Array coercion
Alex Savitsky created NIFI-5731:
---
Summary: Support List-typed fields for Array coercion
Key: NIFI-5731
URL: https://issues.apache.org/jira/browse/NIFI-5731
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 1.7.1
Reporter: Alex Savitsky
This Jira concerns the nifi-record module specifically.
Currently, the toArray method of
org.apache.nifi.serialization.record.util.DataTypeUtils does not support values
of type java.util.List to convert to Object[], despite the conversion being a
trivial one-liner:
{code:java}
return ((List) value).toArray();
{code}
The Record being converted doesn't always have the control to enforce all its
arrays being actual arrays and not collections (e.g., if the Record was created
in Groovy), and it would be nice to have it converted on the fly, rather than
having to transform it manually.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5730) Remove System JS specific strings from template URLs
[
https://issues.apache.org/jira/browse/NIFI-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Horváth updated NIFI-5730:
-
Description:
There is a `!text` string at the end of every Angular component template path
when they are required by it's component.
E.g:
{code:java}
require('./confirm-dialog.component.html!text'){code}
This is a System JS Text plugin specific option and should be only applied when
used with System JS. The issue came around when tried to build the project with
Webpack. It fails to build the project because if this string and it's hard to
tell Webpack to ignore it.
was:
There is a `!text` string at the end of every Angular component template path
when they are required by it's component.
E.g: ```require('./confirm-dialog.component.html!text')```
This is a System JS Text plugin specific option and should be only applied when
used with System JS. The issue came around when tried to build the project with
Webpack. It fails to build the project because if this string and it's hard to
tell Webpack to ignore it.
> Remove System JS specific strings from template URLs
>
>
> Key: NIFI-5730
> URL: https://issues.apache.org/jira/browse/NIFI-5730
> Project: Apache NiFi
> Issue Type: Bug
> Components: FDS
>Reporter: László Horváth
>Priority: Minor
>
> There is a `!text` string at the end of every Angular component template path
> when they are required by it's component.
> E.g:
> {code:java}
> require('./confirm-dialog.component.html!text'){code}
> This is a System JS Text plugin specific option and should be only applied
> when used with System JS. The issue came around when tried to build the
> project with Webpack. It fails to build the project because if this string
> and it's hard to tell Webpack to ignore it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5730) Remove System JS specific strings from template URLs
[
https://issues.apache.org/jira/browse/NIFI-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Horváth updated NIFI-5730:
-
Description:
There is a `!text` string at the end of every Angular component template path
when they are required by it's component.
E.g: ```require('./confirm-dialog.component.html!text')```
This is a System JS Text plugin specific option and should be only applied when
used with System JS. The issue came around when tried to build the project with
Webpack. It fails to build the project because if this string and it's hard to
tell Webpack to ignore it.
was:
There is a `!text` string at the end of every Angular component template path
when they are required by it's component.
E.g: `require('./confirm-dialog.component.html!text')`
This is a System JS Text plugin specific option and should be only applied
when used with System JS. The issue came around when tried to build the project
with Webpack. It fails to build the project because if this string and it's
hard to tell Webpack to ignore it.
> Remove System JS specific strings from template URLs
>
>
> Key: NIFI-5730
> URL: https://issues.apache.org/jira/browse/NIFI-5730
> Project: Apache NiFi
> Issue Type: Bug
> Components: FDS
>Reporter: László Horváth
>Priority: Minor
>
> There is a `!text` string at the end of every Angular component template path
> when they are required by it's component.
> E.g: ```require('./confirm-dialog.component.html!text')```
> This is a System JS Text plugin specific option and should be only applied
> when used with System JS. The issue came around when tried to build the
> project with Webpack. It fails to build the project because if this string
> and it's hard to tell Webpack to ignore it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5730) Remove System JS specific strings from template URLs
László Horváth created NIFI-5730:
Summary: Remove System JS specific strings from template URLs
Key: NIFI-5730
URL: https://issues.apache.org/jira/browse/NIFI-5730
Project: Apache NiFi
Issue Type: Bug
Components: FDS
Reporter: László Horváth
There is a `!text` string at the end of every Angular component template path
when they are required by it's component.
E.g: `require('./confirm-dialog.component.html!text')`
This is a System JS Text plugin specific option and should be only applied
when used with System JS. The issue came around when tried to build the project
with Webpack. It fails to build the project because if this string and it's
hard to tell Webpack to ignore it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5729) Error: "the trustAnchors parameter must be non-empty" occurs when running GetSQS using the endpoint override property
Regine D. created NIFI-5729:
---
Summary: Error: "the trustAnchors parameter must be non-empty"
occurs when running GetSQS using the endpoint override property
Key: NIFI-5729
URL: https://issues.apache.org/jira/browse/NIFI-5729
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.8.0
Reporter: Regine D.
The following error occurred when utilizing the endpoint override property:
{code:java}
Failed to send messages to Amazon SQS due to Unable to execute HTTP request:
java.lang.RuntimeException: Unexpected error:
java.security.InvalidAlgorithParameterException: the trustAnchors parameter
must be non-empty; routing to failure:
com.amazonaws.SdkClientException: Unable to execute HTTP request:
java.lang.RuntimeException: Unexpected error:
java.security.InvalidAlgorithmParameterException: the trustAnchors parameter
must be non-empty{code}
Possible fix, adding SSLContextService to DeleteSQS, GetSQS, PutSQS
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[ https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFICPP-647. --- Resolution: Fixed Assignee: Mr TheSegfault > Deprecate functions in API that pass char* in favor of those more strongly > typed > > > Key: MINIFICPP-647 > URL: https://issues.apache.org/jira/browse/MINIFICPP-647 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Blocker > Labels: CAPI > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[ https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656876#comment-16656876 ] ASF GitHub Bot commented on MINIFICPP-647: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/420 > Deprecate functions in API that pass char* in favor of those more strongly > typed > > > Key: MINIFICPP-647 > URL: https://issues.apache.org/jira/browse/MINIFICPP-647 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Priority: Blocker > Labels: CAPI > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #420: MINIFICPP-647: Begin process of deprecati...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/420 ---
[jira] [Created] (NIFI-5728) Inconsistent behavior in XMLRecordSetWriter for Root Record Tag
Ed Berezitsky created NIFI-5728:
---
Summary: Inconsistent behavior in XMLRecordSetWriter for Root
Record Tag
Key: NIFI-5728
URL: https://issues.apache.org/jira/browse/NIFI-5728
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.7.1
Reporter: Ed Berezitsky
Assignee: Ed Berezitsky
In XMLRecordSetWriter:
When used with "Use Schema Text" and "Name of Record Tag" is empty (so record
name should be used to wrapping XML tag) - works correctly.
When used with Schema Registry and "Name of Record Tag" is empty - it doesn't
write record name, but uses schema name instead. I believe this inconsistency
is coming from the fact that when schema in a writer defined by "Use Schema
Text" - it doesn't have a name and then name is taken from a record. But when
it comes from registry - it simply uses schema idetifier:
{code:java}
recordSchema.getIdentifier().getName();
{code}
IMO root record name should be used in this case instead of schema name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFI-5480) Improve efficiency of how components are looked up by Identifier
[
https://issues.apache.org/jira/browse/NIFI-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Gilman resolved NIFI-5480.
---
Resolution: Fixed
Fix Version/s: 1.7.0
> Improve efficiency of how components are looked up by Identifier
>
>
> Key: NIFI-5480
> URL: https://issues.apache.org/jira/browse/NIFI-5480
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.7.0
>
>
> When we lookup a component by ID, we do so by obtaining the Root Process
> Group and then calling {{findLocalConnectable(String id)}}. This method
> obtains a read lock, then checks its map of Processors, its map of Input
> Ports, its map of Output Ports, and its map of Funnels. If no match is found,
> it then calls getRemoteProcessGroups() to iterate over each of those, looking
> for a Remote Input/Output Port with that ID. This call to
> {{getRemoteProcessGroups()}} creates a new {{HashSet}} that is then returned.
> If no match is found, we then call {{getProcessGroups()}} which also creates
> a new {{HashSet}} of ProcessGroup objects, and we iterate over those
> (recursively).
> This means that for each call to lookup a component by ID, we have to create
> two {{HashSet}}s - for each Process Group on the canvas, until the component
> is found. Consider a flow that has a dozen Process Groups and several
> thousand Processors/ports/funnels. If we then click "Start" on the root
> group, we must create up to 24 {{HashSet}} objects and obtain 12 Read Locks.
> This is done for each component, so for 1,000 Processors it will create
> 24,000 {{HashSet}}s and obtain 12,000 Read Locks. Also, since this is a
> mutable request, this has to be done for both the first and second phase of
> the request, which results in a total of 48,000 {{HashSet}}s and 24,000 Read
> Locks being obtained.
> Testing with 10,000 Processors I am seeing requests take well over 30 seconds
> to complete. All just to find a component by identifier. We can make this
> much more efficient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-647) Deprecate functions in API that pass char* in favor of those more strongly typed
[ https://issues.apache.org/jira/browse/MINIFICPP-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656825#comment-16656825 ] ASF GitHub Bot commented on MINIFICPP-647: -- Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/420 Thanks for updating. Will get this merged in. > Deprecate functions in API that pass char* in favor of those more strongly > typed > > > Key: MINIFICPP-647 > URL: https://issues.apache.org/jira/browse/MINIFICPP-647 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Priority: Blocker > Labels: CAPI > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #420: MINIFICPP-647: Begin process of deprecating func...
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/420 Thanks for updating. Will get this merged in. ---
[jira] [Updated] (NIFI-5727) Optimize GenerateTableFetch to remove unnecessary COUNT(*)
[
https://issues.apache.org/jira/browse/NIFI-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Nikotin updated NIFI-5727:
---
External issue URL: (was: https://github.com/apache/nifi/pull/3094)
> Optimize GenerateTableFetch to remove unnecessary COUNT(*)
> --
>
> Key: NIFI-5727
> URL: https://issues.apache.org/jira/browse/NIFI-5727
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Valentin Nikotin
>Priority: Minor
>
> When processor is configured to use *"Column for Value Partitioning"*
> processor property it still
> [calculates|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java#L293]
> {{COUNT(*)}} while it's not used further in code.
> At line 292 code adds an explicit {{SELECT COUNT(*)}} to the query that
> calculates the bounds of the partitions:
> {code:java}
> List maxValueSelectColumns = new
> ArrayList<>(numMaxValueColumns + 1);
> maxValueSelectColumns.add("COUNT(*)");
> {code}
> At [line
> 414|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java#L414]
> we see that the count is only used if no columns have been set for paging:
> {code:java}
> if (useColumnValsForPaging) {
> final long valueRangeSize = maxValueForPartitioning ==
> null ? 0 : (maxValueForPartitioning - minValueForPartitioning + 1);
> numberOfFetches = (partitionSize == 0) ? 1 :
> (valueRangeSize / partitionSize) + (valueRangeSize % partitionSize == 0 ? 0 :
> 1);
> } else {
> numberOfFetches = (partitionSize == 0) ? 1 : (rowCount /
> partitionSize) + (rowCount % partitionSize == 0 ? 0 : 1);
> }
> {code}
> Since the {{SELECT COUNT(*)}} sometimes takes too long to complete, we want
> to optimize this code and perform the {{SELECT COUNT(*)}} only when property
> *"Column for Value Partitioning"* has not been set.
> The idea is to use *-1* value to minimize code changes:
> {code:java}
> if (useColumnValsForPaging) {
> maxValueSelectColumns.add("-1");
> } else {
> maxValueSelectColumns.add("COUNT(*)");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5727) Optimize GenerateTableFetch to remove unnecessary COUNT(*)
[
https://issues.apache.org/jira/browse/NIFI-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Nikotin updated NIFI-5727:
---
External issue URL: https://github.com/apache/nifi/pull/3094
> Optimize GenerateTableFetch to remove unnecessary COUNT(*)
> --
>
> Key: NIFI-5727
> URL: https://issues.apache.org/jira/browse/NIFI-5727
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Valentin Nikotin
>Priority: Minor
>
> When processor is configured to use *"Column for Value Partitioning"*
> processor property it still
> [calculates|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java#L293]
> {{COUNT(*)}} while it's not used further in code.
> At line 292 code adds an explicit {{SELECT COUNT(*)}} to the query that
> calculates the bounds of the partitions:
> {code:java}
> List maxValueSelectColumns = new
> ArrayList<>(numMaxValueColumns + 1);
> maxValueSelectColumns.add("COUNT(*)");
> {code}
> At [line
> 414|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java#L414]
> we see that the count is only used if no columns have been set for paging:
> {code:java}
> if (useColumnValsForPaging) {
> final long valueRangeSize = maxValueForPartitioning ==
> null ? 0 : (maxValueForPartitioning - minValueForPartitioning + 1);
> numberOfFetches = (partitionSize == 0) ? 1 :
> (valueRangeSize / partitionSize) + (valueRangeSize % partitionSize == 0 ? 0 :
> 1);
> } else {
> numberOfFetches = (partitionSize == 0) ? 1 : (rowCount /
> partitionSize) + (rowCount % partitionSize == 0 ? 0 : 1);
> }
> {code}
> Since the {{SELECT COUNT(*)}} sometimes takes too long to complete, we want
> to optimize this code and perform the {{SELECT COUNT(*)}} only when property
> *"Column for Value Partitioning"* has not been set.
> The idea is to use *-1* value to minimize code changes:
> {code:java}
> if (useColumnValsForPaging) {
> maxValueSelectColumns.add("-1");
> } else {
> maxValueSelectColumns.add("COUNT(*)");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5727) Optimize GenerateTableFetch to remove unnecessary COUNT(*)
[
https://issues.apache.org/jira/browse/NIFI-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656677#comment-16656677
]
ASF GitHub Bot commented on NIFI-5727:
--
GitHub user rednikotin opened a pull request:
https://github.com/apache/nifi/pull/3094
NIFI-5727 - Optimize GenerateTableFetch to remove unnecessary COUNT(*)
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rednikotin/nifi NIFI-5727
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3094.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3094
commit 62b061e91b7e8475603a4bcd1e4d06636824ef1f
Author: rednikotin
Date: 2018-10-19T11:22:33Z
Added replace unnecessary row count with -1 stub value when paging is used
> Optimize GenerateTableFetch to remove unnecessary COUNT(*)
> --
>
> Key: NIFI-5727
> URL: https://issues.apache.org/jira/browse/NIFI-5727
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Valentin Nikotin
>Priority: Minor
>
> When processor is configured to use *"Column for Value Partitioning"*
> processor property it still
> [calculates|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java#L293]
> {{COUNT(*)}} while it's not used further in code.
> At line 292 code adds an explicit {{SELECT COUNT(*)}} to the query that
> calculates the bounds of the partitions:
> {code:java}
> List maxValueSelectColumns = new
> ArrayList<>(numMaxValueColumns + 1);
> maxValueSelectColumns.add("COUNT(*)");
> {code}
> At [line
> 414|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java#L414]
> we see that the count is only used if no columns have been set for paging:
> {code:java}
> if (useColumnValsForPaging) {
> final long valueRangeSize = maxValueForPartitioning ==
> null ? 0 : (maxValueForPartitioning - minValueForPartitioning + 1);
> numberOfFetches = (partitionSize == 0) ? 1 :
> (valueRangeSize / partitionSize) + (valueRangeSize % partitionSize == 0 ? 0 :
> 1);
> } else {
> numberOfFetches = (partitionSize == 0) ? 1 : (rowCount /
> partitionSize) + (rowCount % partitionSize == 0 ? 0 : 1);
> }
> {code}
> Since the {{SELECT COUNT(*)}} sometimes takes too long to complete, we want
> to optimize this code and perform the {{SELECT COUNT(*)}} only when property
> *"Column for Value Partitioning"* has not been set.
> The idea is to use *-1* value to minimize code changes:
> {code:ja
[GitHub] nifi pull request #3094: NIFI-5727 - Optimize GenerateTableFetch to remove u...
GitHub user rednikotin opened a pull request: https://github.com/apache/nifi/pull/3094 NIFI-5727 - Optimize GenerateTableFetch to remove unnecessary COUNT(*) Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rednikotin/nifi NIFI-5727 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3094.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3094 commit 62b061e91b7e8475603a4bcd1e4d06636824ef1f Author: rednikotin Date: 2018-10-19T11:22:33Z Added replace unnecessary row count with -1 stub value when paging is used ---
[jira] [Created] (NIFI-5727) Optimize GenerateTableFetch to remove unnecessary COUNT(*)
Valentin Nikotin created NIFI-5727:
--
Summary: Optimize GenerateTableFetch to remove unnecessary COUNT(*)
Key: NIFI-5727
URL: https://issues.apache.org/jira/browse/NIFI-5727
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 1.7.1
Reporter: Valentin Nikotin
When processor is configured to use *"Column for Value Partitioning"* processor
property it still
[calculates|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java#L293]
{{COUNT(*)}} while it's not used further in code.
At line 292 code adds an explicit {{SELECT COUNT(*)}} to the query that
calculates the bounds of the partitions:
{code:java}
List maxValueSelectColumns = new
ArrayList<>(numMaxValueColumns + 1);
maxValueSelectColumns.add("COUNT(*)");
{code}
At [line
414|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java#L414]
we see that the count is only used if no columns have been set for paging:
{code:java}
if (useColumnValsForPaging) {
final long valueRangeSize = maxValueForPartitioning == null
? 0 : (maxValueForPartitioning - minValueForPartitioning + 1);
numberOfFetches = (partitionSize == 0) ? 1 :
(valueRangeSize / partitionSize) + (valueRangeSize % partitionSize == 0 ? 0 :
1);
} else {
numberOfFetches = (partitionSize == 0) ? 1 : (rowCount /
partitionSize) + (rowCount % partitionSize == 0 ? 0 : 1);
}
{code}
Since the {{SELECT COUNT(*)}} sometimes takes too long to complete, we want to
optimize this code and perform the {{SELECT COUNT(*)}} only when property
*"Column for Value Partitioning"* has not been set.
The idea is to use *-1* value to minimize code changes:
{code:java}
if (useColumnValsForPaging) {
maxValueSelectColumns.add("-1");
} else {
maxValueSelectColumns.add("COUNT(*)");
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5726) Update README with Slack Workspace information
[ https://issues.apache.org/jira/browse/NIFI-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5726: - Status: Patch Available (was: Open) > Update README with Slack Workspace information > -- > > Key: NIFI-5726 > URL: https://issues.apache.org/jira/browse/NIFI-5726 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > Update README with Slack Workspace information -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3093: NIFI-5726 - Update README with Slack Workspace info...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3093 NIFI-5726 - Update README with Slack Workspace information Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5726 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3093.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3093 commit cc7756e33f7d63e018b232180e42b8a94556e84a Author: Pierre Villard Date: 2018-10-19T09:08:08Z NIFI-5726 - Update README with Slack Workspace information ---
[jira] [Commented] (NIFI-5726) Update README with Slack Workspace information
[ https://issues.apache.org/jira/browse/NIFI-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656487#comment-16656487 ] ASF GitHub Bot commented on NIFI-5726: -- GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3093 NIFI-5726 - Update README with Slack Workspace information Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5726 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3093.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3093 commit cc7756e33f7d63e018b232180e42b8a94556e84a Author: Pierre Villard Date: 2018-10-19T09:08:08Z NIFI-5726 - Update README with Slack Workspace information > Update README with Slack Workspace information > -- > > Key: NIFI-5726 > URL: https://issues.apache.org/jira/browse/NIFI-5726 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > Update README with Slack Workspace information -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5726) Update README with Slack Workspace information
Pierre Villard created NIFI-5726: Summary: Update README with Slack Workspace information Key: NIFI-5726 URL: https://issues.apache.org/jira/browse/NIFI-5726 Project: Apache NiFi Issue Type: Improvement Components: Documentation & Website Reporter: Pierre Villard Assignee: Pierre Villard Update README with Slack Workspace information -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5525) CSVRecordReader fails with StringIndexOutOfBoundsException when field is a double quote
[
https://issues.apache.org/jira/browse/NIFI-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5525:
-
Status: Patch Available (was: Reopened)
> CSVRecordReader fails with StringIndexOutOfBoundsException when field is a
> double quote
> ---
>
> Key: NIFI-5525
> URL: https://issues.apache.org/jira/browse/NIFI-5525
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Vadim
>Assignee: Pierre Villard
>Priority: Major
> Labels: easyfix, pull-request-available
> Fix For: 1.8.0
>
>
> *Bug description:*
> When trying to parse a CSV file given in RFC4180 format and one of its fields
> is a double quote, CSVRecordReader fails with the following exception:
> {quote}java.lang.StringIndexOutOfBoundsException: String index out of range:
> -1
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.nifi.csv.AbstractCSVRecordReader.convert(AbstractCSVRecordReader.java:82)
> at org.apache.nifi.csv.CSVRecordReader.nextRecord(CSVRecordReader.java:102)
> at org.apache.nifi.serialization.RecordReader.nextRecord(RecordReader.java:50)
> at
> org.apache.nifi.csv.TestCSVRecordReader.testQuote(TestCSVRecordReader.java:610)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {quote}
>
> Note, that according to RFC4180:
>
> If double-quotes are used to enclose fields, then a double-quote
>appearing inside a field must be escaped by preceding it with
>another double quote.
> [https://tools.ietf.org/html/rfc4180#page-2]
>
> Then a field whose value is a double quote character would be encoded like
> this:
>
> (4 double quote characters)
> *How to reproduce*
> Add the following method to TestCSVRecordReader.java and run the test:
>
> {code:java}
> @Test
> public void testQuote() throws IOException, MalformedRecordException {
> final CSVFormat format =
> CSVFormat.RFC4180.withFirstRecordAsHeader().withTrim().withQuote('"');
> final String text = "\"name\"\n\"\"\"\"";
> final List fields = new ArrayList<>();
> fields.add(new RecordField("name", RecordFieldType.STRING.getDataType()));
> final RecordSchema schema = new SimpleRecordSchema(fields);
> try (final InputStream bais = new
> ByteArrayInputStream(text.getBytes(StandardCharsets.UTF_8));
> final CSVRecordReader reader = new CSVRecordReader(bais,
> Mockito.mock(ComponentLog.class), schema, format, true, false,
> RecordFieldType.DATE.getDefaultFormat(),
> RecordFieldType.TIME.getDefaultFormat(),
> RecordFieldType.TIMESTAMP.getDefaultFormat(), StandardCharsets.UTF_8.name()))
> {
> final Record record = reader.nextRecord();
> final String name = (String)record.getValue("name");
> assertEquals("\"", name);
> }
> }
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5525) CSVRecordReader fails with StringIndexOutOfBoundsException when field is a double quote
[
https://issues.apache.org/jira/browse/NIFI-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656455#comment-16656455
]
ASF GitHub Bot commented on NIFI-5525:
--
GitHub user pvillard31 opened a pull request:
https://github.com/apache/nifi/pull/3092
NIFI-5525 - CSVRecordReader fails with StringIndexOutOfBoundsExceptio…
…n when field is a double quote
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pvillard31/nifi NIFI-5525
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3092.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3092
commit 97ae7f8d7f5c03bf63ebcc7fee74780f9abf7954
Author: Pierre Villard
Date: 2018-10-19T08:44:00Z
NIFI-5525 - CSVRecordReader fails with StringIndexOutOfBoundsException when
field is a double quote
> CSVRecordReader fails with StringIndexOutOfBoundsException when field is a
> double quote
> ---
>
> Key: NIFI-5525
> URL: https://issues.apache.org/jira/browse/NIFI-5525
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Vadim
>Assignee: Pierre Villard
>Priority: Major
> Labels: easyfix, pull-request-available
> Fix For: 1.8.0
>
>
> *Bug description:*
> When trying to parse a CSV file given in RFC4180 format and one of its fields
> is a double quote, CSVRecordReader fails with the following exception:
> {quote}java.lang.StringIndexOutOfBoundsException: String index out of range:
> -1
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.nifi.csv.AbstractCSVRecordReader.convert(AbstractCSVRecordReader.java:82)
> at org.apache.nifi.csv.CSVRecordReader.nextRecord(CSVRecordReader.java:102)
> at org.apache.nifi.serialization.RecordReader.nextRecord(RecordReader.java:50)
> at
> org.apache.nifi.csv.TestCSVRecordReader.testQuote(TestCSVRecordReader.java:610)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runC
[GitHub] nifi pull request #3092: NIFI-5525 - CSVRecordReader fails with StringIndexO...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3092 NIFI-5525 - CSVRecordReader fails with StringIndexOutOfBoundsExceptio⦠â¦n when field is a double quote Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5525 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3092.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3092 commit 97ae7f8d7f5c03bf63ebcc7fee74780f9abf7954 Author: Pierre Villard Date: 2018-10-19T08:44:00Z NIFI-5525 - CSVRecordReader fails with StringIndexOutOfBoundsException when field is a double quote ---
[jira] [Reopened] (NIFI-5525) CSVRecordReader fails with StringIndexOutOfBoundsException when field is a double quote
[
https://issues.apache.org/jira/browse/NIFI-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard reopened NIFI-5525:
--
Assignee: Pierre Villard
Unfortunately one case has not been taken into account and I'm facing the same
issue. Submitting a PR for it.
> CSVRecordReader fails with StringIndexOutOfBoundsException when field is a
> double quote
> ---
>
> Key: NIFI-5525
> URL: https://issues.apache.org/jira/browse/NIFI-5525
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Vadim
>Assignee: Pierre Villard
>Priority: Major
> Labels: easyfix, pull-request-available
> Fix For: 1.8.0
>
>
> *Bug description:*
> When trying to parse a CSV file given in RFC4180 format and one of its fields
> is a double quote, CSVRecordReader fails with the following exception:
> {quote}java.lang.StringIndexOutOfBoundsException: String index out of range:
> -1
> at java.lang.String.substring(String.java:1967)
> at
> org.apache.nifi.csv.AbstractCSVRecordReader.convert(AbstractCSVRecordReader.java:82)
> at org.apache.nifi.csv.CSVRecordReader.nextRecord(CSVRecordReader.java:102)
> at org.apache.nifi.serialization.RecordReader.nextRecord(RecordReader.java:50)
> at
> org.apache.nifi.csv.TestCSVRecordReader.testQuote(TestCSVRecordReader.java:610)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {quote}
>
> Note, that according to RFC4180:
>
> If double-quotes are used to enclose fields, then a double-quote
>appearing inside a field must be escaped by preceding it with
>another double quote.
> [https://tools.ietf.org/html/rfc4180#page-2]
>
> Then a field whose value is a double quote character would be encoded like
> this:
>
> (4 double quote characters)
> *How to reproduce*
> Add the following method to TestCSVRecordReader.java and run the test:
>
> {code:java}
> @Test
> public void testQuote() throws IOException, MalformedRecordException {
> final CSVFormat format =
> CSVFormat.RFC4180.withFirstRecordAsHeader().withTrim().withQuote('"');
> final String text = "\"name\"\n\"\"\"\"";
> final List fields = new ArrayList<>();
> fields.add(new RecordField("name", RecordFieldType.STRING.getDataType()));
> final RecordSchema schema = new SimpleRecordSchema(fields);
> try (final InputStream bais = new
> ByteArrayInputStream(text.getBytes(StandardCharsets.UTF_8));
> final CSVRecordReader reader = new CSVRecordReader(bais,
> Mockito.mock(ComponentLog.class), schema, format, true, false,
> RecordFieldType.DATE.getDefaultFormat(),
> RecordFieldType.TIME.getDefaultFormat(),
> RecordFieldType.TIMESTAMP.getDefaultFormat(), StandardCharsets.UTF_8.name()))
> {
> final Record record = reader.nextRecord();
> final String name = (String)record.getValue("name");
> assertEquals("\"", name);
> }
> }
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
