[jira] [Commented] (NIFI-2702) GetJMS vs ConsumeJMS
[ https://issues.apache.org/jira/browse/NIFI-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024298#comment-16024298 ] ASF GitHub Bot commented on NIFI-2702: -- Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1643 +1, merged to master, thanks @trixpan ! > GetJMS vs ConsumeJMS > > > Key: NIFI-2702 > URL: https://issues.apache.org/jira/browse/NIFI-2702 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Affects Versions: 1.0.0 >Reporter: Raymond >Assignee: Andre F de Miranda >Priority: Trivial > Fix For: 1.3.0 > > > In the GetJMS processors (Queue and Topic) you only can choose ActiveMQ as > the MQ provider. In the documentation there is no indication that > 1)ActiveMQ the only supported provider is > 2)How to use another provider > 3)That for other MQ providers one should use ConsumeJMS with a > controlling service > 4)That this processor is deprecated (as Oleg said on the developer list > in May > http://apache-nifi-developer-list.39713.n7.nabble.com/NIFI-amp-IBM-MQ-td10651.html) > Maybe good to make this clearer in the documentation -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-2702) GetJMS vs ConsumeJMS
[ https://issues.apache.org/jira/browse/NIFI-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024297#comment-16024297 ] ASF GitHub Bot commented on NIFI-2702: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1643 > GetJMS vs ConsumeJMS > > > Key: NIFI-2702 > URL: https://issues.apache.org/jira/browse/NIFI-2702 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Affects Versions: 1.0.0 >Reporter: Raymond >Assignee: Andre F de Miranda >Priority: Trivial > Fix For: 1.3.0 > > > In the GetJMS processors (Queue and Topic) you only can choose ActiveMQ as > the MQ provider. In the documentation there is no indication that > 1)ActiveMQ the only supported provider is > 2)How to use another provider > 3)That for other MQ providers one should use ConsumeJMS with a > controlling service > 4)That this processor is deprecated (as Oleg said on the developer list > in May > http://apache-nifi-developer-list.39713.n7.nabble.com/NIFI-amp-IBM-MQ-td10651.html) > Maybe good to make this clearer in the documentation -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-2702) GetJMS vs ConsumeJMS
[ https://issues.apache.org/jira/browse/NIFI-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-2702: - Resolution: Fixed Fix Version/s: 1.3.0 Status: Resolved (was: Patch Available) > GetJMS vs ConsumeJMS > > > Key: NIFI-2702 > URL: https://issues.apache.org/jira/browse/NIFI-2702 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Affects Versions: 1.0.0 >Reporter: Raymond >Assignee: Andre F de Miranda >Priority: Trivial > Fix For: 1.3.0 > > > In the GetJMS processors (Queue and Topic) you only can choose ActiveMQ as > the MQ provider. In the documentation there is no indication that > 1)ActiveMQ the only supported provider is > 2)How to use another provider > 3)That for other MQ providers one should use ConsumeJMS with a > controlling service > 4)That this processor is deprecated (as Oleg said on the developer list > in May > http://apache-nifi-developer-list.39713.n7.nabble.com/NIFI-amp-IBM-MQ-td10651.html) > Maybe good to make this clearer in the documentation -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1643: NIFI-2702 - Deprecates nifi-standard-bundle/*JMS and docum...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1643 +1, merged to master, thanks @trixpan ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-2702) GetJMS vs ConsumeJMS
[ https://issues.apache.org/jira/browse/NIFI-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024296#comment-16024296 ] ASF subversion and git services commented on NIFI-2702: --- Commit c07850aec3a9017ffd35d571fcb4324396de09d9 in nifi's branch refs/heads/master from [~trixpan] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=c07850a ] NIFI-2702 - Deprecates nifi-standard-bundle/*JMS and document the recommended alternative Signed-off-by: Pierre Villard This closes #1643. > GetJMS vs ConsumeJMS > > > Key: NIFI-2702 > URL: https://issues.apache.org/jira/browse/NIFI-2702 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Affects Versions: 1.0.0 >Reporter: Raymond >Assignee: Andre F de Miranda >Priority: Trivial > > In the GetJMS processors (Queue and Topic) you only can choose ActiveMQ as > the MQ provider. In the documentation there is no indication that > 1)ActiveMQ the only supported provider is > 2)How to use another provider > 3)That for other MQ providers one should use ConsumeJMS with a > controlling service > 4)That this processor is deprecated (as Oleg said on the developer list > in May > http://apache-nifi-developer-list.39713.n7.nabble.com/NIFI-amp-IBM-MQ-td10651.html) > Maybe good to make this clearer in the documentation -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1643: NIFI-2702 - Deprecates nifi-standard-bundle/*JMS an...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1643 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3809) S2S Reporting tasks should provide the option to choose transport protocol
[ https://issues.apache.org/jira/browse/NIFI-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024283#comment-16024283 ] ASF GitHub Bot commented on NIFI-3809: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1754 > S2S Reporting tasks should provide the option to choose transport protocol > -- > > Key: NIFI-3809 > URL: https://issues.apache.org/jira/browse/NIFI-3809 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.2.0 >Reporter: Pierre Villard >Assignee: Pierre Villard > > Right now Site-to-Site based reporting tasks are using the default RAW > protocol but some users might not want to open a port for that. It should be > possible to use HTTP. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1754: NIFI-3809 - Added HTTP mode and HTTP proxy for S2S ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1754 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3809) S2S Reporting tasks should provide the option to choose transport protocol
[ https://issues.apache.org/jira/browse/NIFI-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024281#comment-16024281 ] ASF subversion and git services commented on NIFI-3809: --- Commit e05005584d52b560771a0ccf2766ee9ee92b6518 in nifi's branch refs/heads/master from [~pvillard] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=e050055 ] NIFI-3809 - Added HTTP mode and HTTP proxy for S2S Reporting Tasks This closes #1754. Signed-off-by: Koji Kawamura > S2S Reporting tasks should provide the option to choose transport protocol > -- > > Key: NIFI-3809 > URL: https://issues.apache.org/jira/browse/NIFI-3809 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.2.0 >Reporter: Pierre Villard >Assignee: Pierre Villard > > Right now Site-to-Site based reporting tasks are using the default RAW > protocol but some users might not want to open a port for that. It should be > possible to use HTTP. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3809) S2S Reporting tasks should provide the option to choose transport protocol
[ https://issues.apache.org/jira/browse/NIFI-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024275#comment-16024275 ] ASF GitHub Bot commented on NIFI-3809: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1754 Thanks for the update @pvillard31 . I confirmed Proxy can be specified with all three reporting tasks. LGTM +1. I'm going to squash commits and merge into master. Thank you! > S2S Reporting tasks should provide the option to choose transport protocol > -- > > Key: NIFI-3809 > URL: https://issues.apache.org/jira/browse/NIFI-3809 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.2.0 >Reporter: Pierre Villard >Assignee: Pierre Villard > > Right now Site-to-Site based reporting tasks are using the default RAW > protocol but some users might not want to open a port for that. It should be > possible to use HTTP. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1754: NIFI-3809 - Added HTTP mode and HTTP proxy for S2S Reporti...
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1754 Thanks for the update @pvillard31 . I confirmed Proxy can be specified with all three reporting tasks. LGTM +1. I'm going to squash commits and merge into master. Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3484) GenerateTableFetch Should Allow for Right Boundary
[ https://issues.apache.org/jira/browse/NIFI-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024269#comment-16024269 ] ASF GitHub Bot commented on NIFI-3484: -- Github user ilyatau commented on the issue: https://github.com/apache/nifi/pull/1513 It's cool. My question was about when this commits will be in master branch? because our architecture very needs this functionality. Thanks > GenerateTableFetch Should Allow for Right Boundary > -- > > Key: NIFI-3484 > URL: https://issues.apache.org/jira/browse/NIFI-3484 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.2.0 >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Minor > > When using GenerateTableFetch it places no right hand boundary on pages of > data. This can lead to issues when the statement says to get the next 1000 > records greater then a specific key, but records were added to the table > between the time the processor executed and when the SQL is being executed. > As a result it pulls in records that did not exist when the processor was > run. On the next execution of the processor these records will be pulled in > a second time. > Example: > Partition Size = 1000 > First run (no state): Count(*)=4700 and MAX(ID)=4700. > 5 FlowFiles are generated, the last one will say to fetch 1000, not 700. (But > I don't think this is really a bug, just an observation). > 5 Flow Files are now in queue to be executed by ExecuteSQL. Before the 5th > file can execute 400 new rows are added to the table. When the final SQL > statement is executed 300 extra records, with higher ID values, will also be > pulled into NiFi. > Second run (state: ID=4700). Count(*) ID>4700 = 400 and MAX(ID)=5100. > 1 Flow File is generated, but includes 300 records already pulled into NiFI. > The solution is to have an optional property that will let users use the new > MAX(ID) as a right boundary when generating queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1513: NIFI-3484 GenerateTableFetch Should Allow for Right Bounda...
Github user ilyatau commented on the issue: https://github.com/apache/nifi/pull/1513 It's cool. My question was about when this commits will be in master branch? because our architecture very needs this functionality. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024267#comment-16024267 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1830 @jfrazee @mattyb149 I reviewed the updated commit and found few points to improve. I was trying to comment all of those, but since it's easier to explain with code instead of comment, I created another PR based on this one. @jfrazee Would you take a look on #1856 to see if those changes are reasonable? Please find the [commit comments](https://github.com/apache/nifi/pull/1856/commits/4f81af0961c3d65c4df37b9de3934c8fed1a70ff) for what I've changed. If my additional commit looks ok, then I'd give +1 to this PR. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1830: NIFI-3404 Add lookup processor for enrichments/joins to re...
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1830 @jfrazee @mattyb149 I reviewed the updated commit and found few points to improve. I was trying to comment all of those, but since it's easier to explain with code instead of comment, I created another PR based on this one. @jfrazee Would you take a look on #1856 to see if those changes are reasonable? Please find the [commit comments](https://github.com/apache/nifi/pull/1856/commits/4f81af0961c3d65c4df37b9de3934c8fed1a70ff) for what I've changed. If my additional commit looks ok, then I'd give +1 to this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024259#comment-16024259 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118425516 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- I think supporting EL is useful, because user can refer variable registries or do some text processing with EL. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118425516 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- I think supporting EL is useful, because user can refer variable registries or do some text processing with EL. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura updated NIFI-3404: Status: Patch Available (was: Open) > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1794: NIFI-1709 - Introduce logic to probe Linux version using /...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1794 @jfrazee sorry. I misread your comment. It works on leap?! Great. :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024250#comment-16024250 ] ASF GitHub Bot commented on NIFI-3404: -- GitHub user ijokarumawak opened a pull request: https://github.com/apache/nifi/pull/1856 NIFI-3404: LookupAttribute, SimpleCsvFileLookupService, PropertiesFileLookupService, XMLFileLookupService Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijokarumawak/nifi nifi-3404 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1856.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1856 commit 86d4198ce5537387a375e23003c8265be2c1e14d Author: Joey Frazee Date: 2017-05-19T16:45:14Z NIFI-3404 Added LookupAttribute processor and lookup controller services commit 4f81af0961c3d65c4df37b9de3934c8fed1a70ff Author: Koji Kawamura Date: 2017-05-25T04:22:54Z NIFI-3404: Improved UX of LookupAttributes. - Added dependency notice. - Added EL evaluation at SimpleKeyValueLookupService. - Updated documentation. - Updated CommonsConfigurationLookupService to throw LookupFailureException if it fails to get configuration so that error messages can be displayed at each processor bulletin. - Added calling getConfiguration at OnEnabled of CommonsConfigurationLookupService, so that the service will stay in Enabling state if there is any issue. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-1709) The nifi.sh install script assumes RHEL directories
[ https://issues.apache.org/jira/browse/NIFI-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024251#comment-16024251 ] ASF GitHub Bot commented on NIFI-1709: -- Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1794 @jfrazee sorry. I misread your comment. It works on leap?! Great. :-) > The nifi.sh install script assumes RHEL directories > --- > > Key: NIFI-1709 > URL: https://issues.apache.org/jira/browse/NIFI-1709 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build > Environment: SUSE >Reporter: David A. Wynne >Assignee: Andre F de Miranda >Priority: Minor > > When setting up NiFi, the command: > bin/nifi.sh install > The following error occurs: > ln: failed to create symbolic link `/etc/rc2.d/S65nifi': No such file or > directory > ln: failed to create symbolic link `/etc/rc2.d/K65nifi': No such file or > directory > Service nifi installed > Looking in the nifi.sh, around line 145 - 148, you see: > rm -f "/etc/rc2.d/S65${SVC_NAME}" > ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/S65${SVC_NAME}" > rm -f "/etc/rc2.d/K65${SVC_NAME}" > ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" > It tries to symlink from /etc/init.d/nifi to /etc/rc2.d/S65nifi (and > K65nifi). > The problem is that the script assumes that /etc/rc2.d exists in a SUSE > system, which it doesn't. > In Suse11, this directory is /etc/init.d/rc2.d/ > The script, especially the "install" option should correctly identify the OS > flavor it runs on, and install the files in the correct location. > The next version of RHEL (8, not out yet), and Ubuntu 16.02 LTS (releasing > next month), will use systemD, which does not have the concept of /etc/rc2.d > directories, but works with "targets". The script will not be compatible with > these OS changes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1856: NIFI-3404: LookupAttribute, SimpleCsvFileLookupServ...
GitHub user ijokarumawak opened a pull request: https://github.com/apache/nifi/pull/1856 NIFI-3404: LookupAttribute, SimpleCsvFileLookupService, PropertiesFileLookupService, XMLFileLookupService Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijokarumawak/nifi nifi-3404 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1856.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1856 commit 86d4198ce5537387a375e23003c8265be2c1e14d Author: Joey Frazee Date: 2017-05-19T16:45:14Z NIFI-3404 Added LookupAttribute processor and lookup controller services commit 4f81af0961c3d65c4df37b9de3934c8fed1a70ff Author: Koji Kawamura Date: 2017-05-25T04:22:54Z NIFI-3404: Improved UX of LookupAttributes. - Added dependency notice. - Added EL evaluation at SimpleKeyValueLookupService. - Updated documentation. - Updated CommonsConfigurationLookupService to throw LookupFailureException if it fails to get configuration so that error messages can be displayed at each processor bulletin. - Added calling getConfiguration at OnEnabled of CommonsConfigurationLookupService, so that the service will stay in Enabling state if there is any issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-1709) The nifi.sh install script assumes RHEL directories
[ https://issues.apache.org/jira/browse/NIFI-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024248#comment-16024248 ] ASF GitHub Bot commented on NIFI-1709: -- Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1794 @jfrazee thanks. I haven't tested on leap. as not working you mean the file is not being linked or the service isn't being started? Cheers > The nifi.sh install script assumes RHEL directories > --- > > Key: NIFI-1709 > URL: https://issues.apache.org/jira/browse/NIFI-1709 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build > Environment: SUSE >Reporter: David A. Wynne >Assignee: Andre F de Miranda >Priority: Minor > > When setting up NiFi, the command: > bin/nifi.sh install > The following error occurs: > ln: failed to create symbolic link `/etc/rc2.d/S65nifi': No such file or > directory > ln: failed to create symbolic link `/etc/rc2.d/K65nifi': No such file or > directory > Service nifi installed > Looking in the nifi.sh, around line 145 - 148, you see: > rm -f "/etc/rc2.d/S65${SVC_NAME}" > ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/S65${SVC_NAME}" > rm -f "/etc/rc2.d/K65${SVC_NAME}" > ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" > It tries to symlink from /etc/init.d/nifi to /etc/rc2.d/S65nifi (and > K65nifi). > The problem is that the script assumes that /etc/rc2.d exists in a SUSE > system, which it doesn't. > In Suse11, this directory is /etc/init.d/rc2.d/ > The script, especially the "install" option should correctly identify the OS > flavor it runs on, and install the files in the correct location. > The next version of RHEL (8, not out yet), and Ubuntu 16.02 LTS (releasing > next month), will use systemD, which does not have the concept of /etc/rc2.d > directories, but works with "targets". The script will not be compatible with > these OS changes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1794: NIFI-1709 - Introduce logic to probe Linux version using /...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1794 @jfrazee thanks. I haven't tested on leap. as not working you mean the file is not being linked or the service isn't being started? Cheers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3484) GenerateTableFetch Should Allow for Right Boundary
[ https://issues.apache.org/jira/browse/NIFI-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024236#comment-16024236 ] ASF GitHub Bot commented on NIFI-3484: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/1513 @ilyatau Why do you feel that this request isn't finished? While I do have a newer version, the functionality it adds is a bit more advanced and I've had a hard time automating the testing. The additional functionality I have working will generate a premature right boundary to reduce the number of records brought back. You might think that this is already the point of Generate Table Fetch, but I found that even with an indexed timestamp column you still have to page the index when paging through data on some systems. In one case we found that on SAP Hana we ran into some internal limitations which did not allow you to page past more than 2 billion rows of data. The table we were loading with Generate Table Fetch had ~6 billion rows. Using the un-committed change you can provide a per execution limit to Generate Table Fetch so that it will only generate pages of `x` size for the first `y` rows in the table. When I tried to test this on SQL Lite it did not work, though it is working on other SQL systems we've tried it on. > GenerateTableFetch Should Allow for Right Boundary > -- > > Key: NIFI-3484 > URL: https://issues.apache.org/jira/browse/NIFI-3484 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.2.0 >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Minor > > When using GenerateTableFetch it places no right hand boundary on pages of > data. This can lead to issues when the statement says to get the next 1000 > records greater then a specific key, but records were added to the table > between the time the processor executed and when the SQL is being executed. > As a result it pulls in records that did not exist when the processor was > run. On the next execution of the processor these records will be pulled in > a second time. > Example: > Partition Size = 1000 > First run (no state): Count(*)=4700 and MAX(ID)=4700. > 5 FlowFiles are generated, the last one will say to fetch 1000, not 700. (But > I don't think this is really a bug, just an observation). > 5 Flow Files are now in queue to be executed by ExecuteSQL. Before the 5th > file can execute 400 new rows are added to the table. When the final SQL > statement is executed 300 extra records, with higher ID values, will also be > pulled into NiFi. > Second run (state: ID=4700). Count(*) ID>4700 = 400 and MAX(ID)=5100. > 1 Flow File is generated, but includes 300 records already pulled into NiFI. > The solution is to have an optional property that will let users use the new > MAX(ID) as a right boundary when generating queries. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1513: NIFI-3484 GenerateTableFetch Should Allow for Right Bounda...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/1513 @ilyatau Why do you feel that this request isn't finished? While I do have a newer version, the functionality it adds is a bit more advanced and I've had a hard time automating the testing. The additional functionality I have working will generate a premature right boundary to reduce the number of records brought back. You might think that this is already the point of Generate Table Fetch, but I found that even with an indexed timestamp column you still have to page the index when paging through data on some systems. In one case we found that on SAP Hana we ran into some internal limitations which did not allow you to page past more than 2 billion rows of data. The table we were loading with Generate Table Fetch had ~6 billion rows. Using the un-committed change you can provide a per execution limit to Generate Table Fetch so that it will only generate pages of `x` size for the first `y` rows in the table. When I tried to test this on SQL Lite it did not work, though it is working on other SQL systems we've tried it on. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1777: NIFI-3859 - Provide filtering options in S2SProvenanceRepo...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1777 Thanks @ijokarumawak ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (NIFI-3859) Provide filtering options in S2SProvenanceReportingTask
[ https://issues.apache.org/jira/browse/NIFI-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura updated NIFI-3859: Resolution: Fixed Fix Version/s: 1.3.0 Status: Resolved (was: Patch Available) > Provide filtering options in S2SProvenanceReportingTask > --- > > Key: NIFI-3859 > URL: https://issues.apache.org/jira/browse/NIFI-3859 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Minor > Fix For: 1.3.0 > > > Based on what will be used the S2SProvenanceReportingTask for, it could be > nice to provide some filtering options to the user in order to generate flow > files only for a subset of the last provenance events. > Filters could be: > - Event type > - Component type > - Component ID -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3859) Provide filtering options in S2SProvenanceReportingTask
[ https://issues.apache.org/jira/browse/NIFI-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024204#comment-16024204 ] ASF subversion and git services commented on NIFI-3859: --- Commit b6eb0ac0fb382b5cc2e3f60196e36835437883f3 in nifi's branch refs/heads/master from [~pvillard] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b6eb0ac ] NIFI-3859 - Provide filtering options in S2SProvenanceReportingTask This closes #1777. Signed-off-by: Koji Kawamura > Provide filtering options in S2SProvenanceReportingTask > --- > > Key: NIFI-3859 > URL: https://issues.apache.org/jira/browse/NIFI-3859 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Minor > > Based on what will be used the S2SProvenanceReportingTask for, it could be > nice to provide some filtering options to the user in order to generate flow > files only for a subset of the last provenance events. > Filters could be: > - Event type > - Component type > - Component ID -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3859) Provide filtering options in S2SProvenanceReportingTask
[ https://issues.apache.org/jira/browse/NIFI-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024205#comment-16024205 ] ASF GitHub Bot commented on NIFI-3859: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1777 > Provide filtering options in S2SProvenanceReportingTask > --- > > Key: NIFI-3859 > URL: https://issues.apache.org/jira/browse/NIFI-3859 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Minor > > Based on what will be used the S2SProvenanceReportingTask for, it could be > nice to provide some filtering options to the user in order to generate flow > files only for a subset of the last provenance events. > Filters could be: > - Event type > - Component type > - Component ID -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1777: NIFI-3859 - Provide filtering options in S2SProvena...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1777 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1777: NIFI-3859 - Provide filtering options in S2SProvena...
Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1777#discussion_r118419716 --- Diff: nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java --- @@ -67,14 +73,73 @@ static final PropertyDescriptor PLATFORM = new PropertyDescriptor.Builder() .name("Platform") +.displayName("Platform") .description("The value to use for the platform field in each provenance event.") .required(true) .expressionLanguageSupported(true) .defaultValue("nifi") .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) .build(); +static final PropertyDescriptor FILTER_EVENT_TYPE = new PropertyDescriptor.Builder() --- End diff -- These newly added properties are not added to `getSupportedPropertyDescriptors` method. So I was not able to configure filters from NIFI UI. Since this is a simple fix, I went ahead and added these filters and able to confirm provenance events are filtered. LGTM, +1. I will merge this into master. Thanks @pvillard31 ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3859) Provide filtering options in S2SProvenanceReportingTask
[ https://issues.apache.org/jira/browse/NIFI-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024203#comment-16024203 ] ASF GitHub Bot commented on NIFI-3859: -- Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1777#discussion_r118419716 --- Diff: nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java --- @@ -67,14 +73,73 @@ static final PropertyDescriptor PLATFORM = new PropertyDescriptor.Builder() .name("Platform") +.displayName("Platform") .description("The value to use for the platform field in each provenance event.") .required(true) .expressionLanguageSupported(true) .defaultValue("nifi") .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) .build(); +static final PropertyDescriptor FILTER_EVENT_TYPE = new PropertyDescriptor.Builder() --- End diff -- These newly added properties are not added to `getSupportedPropertyDescriptors` method. So I was not able to configure filters from NIFI UI. Since this is a simple fix, I went ahead and added these filters and able to confirm provenance events are filtered. LGTM, +1. I will merge this into master. Thanks @pvillard31 ! > Provide filtering options in S2SProvenanceReportingTask > --- > > Key: NIFI-3859 > URL: https://issues.apache.org/jira/browse/NIFI-3859 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Minor > > Based on what will be used the S2SProvenanceReportingTask for, it could be > nice to provide some filtering options to the user in order to generate flow > files only for a subset of the last provenance events. > Filters could be: > - Event type > - Component type > - Component ID -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-1709) The nifi.sh install script assumes RHEL directories
[ https://issues.apache.org/jira/browse/NIFI-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024169#comment-16024169 ] ASF GitHub Bot commented on NIFI-1709: -- Github user jfrazee commented on the issue: https://github.com/apache/nifi/pull/1794 @trixpan Fair enough. Seems clear the lsb_release suggestion was bad. BTW, I tested your changes on opensuse:latest (leap) and it's working as expected. > The nifi.sh install script assumes RHEL directories > --- > > Key: NIFI-1709 > URL: https://issues.apache.org/jira/browse/NIFI-1709 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build > Environment: SUSE >Reporter: David A. Wynne >Assignee: Andre F de Miranda >Priority: Minor > > When setting up NiFi, the command: > bin/nifi.sh install > The following error occurs: > ln: failed to create symbolic link `/etc/rc2.d/S65nifi': No such file or > directory > ln: failed to create symbolic link `/etc/rc2.d/K65nifi': No such file or > directory > Service nifi installed > Looking in the nifi.sh, around line 145 - 148, you see: > rm -f "/etc/rc2.d/S65${SVC_NAME}" > ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/S65${SVC_NAME}" > rm -f "/etc/rc2.d/K65${SVC_NAME}" > ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" > It tries to symlink from /etc/init.d/nifi to /etc/rc2.d/S65nifi (and > K65nifi). > The problem is that the script assumes that /etc/rc2.d exists in a SUSE > system, which it doesn't. > In Suse11, this directory is /etc/init.d/rc2.d/ > The script, especially the "install" option should correctly identify the OS > flavor it runs on, and install the files in the correct location. > The next version of RHEL (8, not out yet), and Ubuntu 16.02 LTS (releasing > next month), will use systemD, which does not have the concept of /etc/rc2.d > directories, but works with "targets". The script will not be compatible with > these OS changes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1794: NIFI-1709 - Introduce logic to probe Linux version using /...
Github user jfrazee commented on the issue: https://github.com/apache/nifi/pull/1794 @trixpan Fair enough. Seems clear the lsb_release suggestion was bad. BTW, I tested your changes on opensuse:latest (leap) and it's working as expected. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3640) GetEventHub should not assume domain names
[ https://issues.apache.org/jira/browse/NIFI-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024119#comment-16024119 ] ASF GitHub Bot commented on NIFI-3640: -- Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/1617 Reviewing... > GetEventHub should not assume domain names > -- > > Key: NIFI-3640 > URL: https://issues.apache.org/jira/browse/NIFI-3640 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Joseph Niemiec >Assignee: Joseph Niemiec > > Today the GetEventHubProcessor hard codes in the servicebus host from Azure > as ".servicebus.windows.net" this does not allow us to deploy to other Geo's > where the URL changes such as "servicebus.chinacloudapi.cn" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1617: NIFI-3640 uri eventhub changes
Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/1617 Reviewing... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user jfrazee commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118408976 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- I meant to remove the expressionLanguageSupported(true). I don't think I had an especially good reason to have added the EL support to the dynamic property for the CS in the first place. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024105#comment-16024105 ] ASF GitHub Bot commented on NIFI-3404: -- Github user jfrazee commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118408976 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- I meant to remove the expressionLanguageSupported(true). I don't think I had an especially good reason to have added the EL support to the dynamic property for the CS in the first place. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024094#comment-16024094 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118408138 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- `cacheConfiguredValues` is still not evaluating expression. I can add EL evaluation when I merge. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-1709) The nifi.sh install script assumes RHEL directories
[ https://issues.apache.org/jira/browse/NIFI-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024093#comment-16024093 ] ASF GitHub Bot commented on NIFI-1709: -- Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1794 @apiri SuSE may or may not be using systemd. Reason why we rely on systemd support to init scripts to achieve the outcome. @jfrazee disagree regarding the use of `lsb_release`: it is an external dependency on most RPM based releases[1][2] and is known to cause issues on systems like Gentoo and archlinux. Same can be said about chkconfig. [1] https://www.unixmen.com/linux-troubleshooting-fix-lsb_release-command-found-centos/ [2] http://0pointer.de/blog/projects/os-release.html > The nifi.sh install script assumes RHEL directories > --- > > Key: NIFI-1709 > URL: https://issues.apache.org/jira/browse/NIFI-1709 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build > Environment: SUSE >Reporter: David A. Wynne >Assignee: Andre F de Miranda >Priority: Minor > > When setting up NiFi, the command: > bin/nifi.sh install > The following error occurs: > ln: failed to create symbolic link `/etc/rc2.d/S65nifi': No such file or > directory > ln: failed to create symbolic link `/etc/rc2.d/K65nifi': No such file or > directory > Service nifi installed > Looking in the nifi.sh, around line 145 - 148, you see: > rm -f "/etc/rc2.d/S65${SVC_NAME}" > ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/S65${SVC_NAME}" > rm -f "/etc/rc2.d/K65${SVC_NAME}" > ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" > It tries to symlink from /etc/init.d/nifi to /etc/rc2.d/S65nifi (and > K65nifi). > The problem is that the script assumes that /etc/rc2.d exists in a SUSE > system, which it doesn't. > In Suse11, this directory is /etc/init.d/rc2.d/ > The script, especially the "install" option should correctly identify the OS > flavor it runs on, and install the files in the correct location. > The next version of RHEL (8, not out yet), and Ubuntu 16.02 LTS (releasing > next month), will use systemD, which does not have the concept of /etc/rc2.d > directories, but works with "targets". The script will not be compatible with > these OS changes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118408138 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java --- @@ -47,6 +47,7 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String .required(false) .dynamic(true) .addValidator(Validator.VALID) +.expressionLanguageSupported(true) --- End diff -- `cacheConfiguredValues` is still not evaluating expression. I can add EL evaluation when I merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1794: NIFI-1709 - Introduce logic to probe Linux version using /...
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1794 @apiri SuSE may or may not be using systemd. Reason why we rely on systemd support to init scripts to achieve the outcome. @jfrazee disagree regarding the use of `lsb_release`: it is an external dependency on most RPM based releases[1][2] and is known to cause issues on systems like Gentoo and archlinux. Same can be said about chkconfig. [1] https://www.unixmen.com/linux-troubleshooting-fix-lsb_release-command-found-centos/ [2] http://0pointer.de/blog/projects/os-release.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning
[ https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024059#comment-16024059 ] ASF subversion and git services commented on NIFI-3971: --- Commit eaefec6d81f9dbf0d239776ef465b14109e28d18 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=eaefec6 ] NIFI-3971: This closes #1854. Fixed bug in calculating content size that was transferred when cloning a relationship Signed-off-by: joewitt > UpdateAttribute metric for output 'size' is larger than it should be when > cloning > - > > Key: NIFI-3971 > URL: https://issues.apache.org/jira/browse/NIFI-3971 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.3.0 > > > When I configure UpdateAttribute so that the 'success' relationship goes to > multiple destinations, the output count is double what it should be. For > instance, if i send in 1 GB of data, the output size should be 2 GB (since > it's cloned to a second connection) but it shows 4 GB. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning
[ https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt resolved NIFI-3971. --- Resolution: Fixed Fix Version/s: 1.3.0 +1 merged to master > UpdateAttribute metric for output 'size' is larger than it should be when > cloning > - > > Key: NIFI-3971 > URL: https://issues.apache.org/jira/browse/NIFI-3971 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.3.0 > > > When I configure UpdateAttribute so that the 'success' relationship goes to > multiple destinations, the output count is double what it should be. For > instance, if i send in 1 GB of data, the output size should be 2 GB (since > it's cloned to a second connection) but it shows 4 GB. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning
[ https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024060#comment-16024060 ] ASF GitHub Bot commented on NIFI-3971: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1854 > UpdateAttribute metric for output 'size' is larger than it should be when > cloning > - > > Key: NIFI-3971 > URL: https://issues.apache.org/jira/browse/NIFI-3971 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.3.0 > > > When I configure UpdateAttribute so that the 'success' relationship goes to > multiple destinations, the output count is double what it should be. For > instance, if i send in 1 GB of data, the output size should be 2 GB (since > it's cloned to a second connection) but it shows 4 GB. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1854: NIFI-3971: Fixed bug in calculating content size th...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1854 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning
[ https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024058#comment-16024058 ] ASF GitHub Bot commented on NIFI-3971: -- Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/1854 +1 will merge to master. thanks > UpdateAttribute metric for output 'size' is larger than it should be when > cloning > - > > Key: NIFI-3971 > URL: https://issues.apache.org/jira/browse/NIFI-3971 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > > When I configure UpdateAttribute so that the 'success' relationship goes to > multiple destinations, the output count is double what it should be. For > instance, if i send in 1 GB of data, the output size should be 2 GB (since > it's cloned to a second connection) but it shows 4 GB. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1854: NIFI-3971: Fixed bug in calculating content size that was ...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/1854 +1 will merge to master. thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart
[ https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-3972: -- Priority: Critical (was: Major) > Controller Service failing to enabled because the service it depends on is > not fully enabled on nifi restart > > > Key: NIFI-3972 > URL: https://issues.apache.org/jira/browse/NIFI-3972 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Critical > Fix For: 1.3.0 > > > I have a JsonRecordSetWriter controller service. It depends on a Schema > Registry controller service. Both are enabled. Upon restart, the > JsonRecordSetWriter service fails to enable, indicating that the Schema > Registry service is not enabled. When I look at the Controller Service > configuration, though, the Schema Registry service is enabled. So it appears > that the JsonRecordSetWriter was attempting to be enabled before the Schema > Registry Service finishes enabling. > {code} > 2017-05-24 19:46:56,314 ERROR [main] > o.a.n.c.s.StandardControllerServiceProvider Failed to enable > JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536] > java.lang.IllegalStateException: Cannot invoke method public abstract > java.util.Set > org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields() > on Controller Service with identifier 0660348b-255e-1f3a--fb399783 > because the Controller Service is disabled > at > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84) > at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source) > at > org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42) > at > org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147) > at > org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149) > at > org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128) > at > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160) > at > org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126) > at > org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326) > at > org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441) > at > org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350) > at > org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152) > at > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042) > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312) > at > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554) > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) > at > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722) > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452) > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800) > at org.apache.nifi.NiFi.(NiFi.java:160) > at org.apache.nifi.NiFi.main(NiFi.java:267) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart
[ https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024054#comment-16024054 ] Joseph Witt commented on NIFI-3972: --- setting fix version to 1.3.0. the pr is close and this is a really important lifecycle condition to resolve > Controller Service failing to enabled because the service it depends on is > not fully enabled on nifi restart > > > Key: NIFI-3972 > URL: https://issues.apache.org/jira/browse/NIFI-3972 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.3.0 > > > I have a JsonRecordSetWriter controller service. It depends on a Schema > Registry controller service. Both are enabled. Upon restart, the > JsonRecordSetWriter service fails to enable, indicating that the Schema > Registry service is not enabled. When I look at the Controller Service > configuration, though, the Schema Registry service is enabled. So it appears > that the JsonRecordSetWriter was attempting to be enabled before the Schema > Registry Service finishes enabling. > {code} > 2017-05-24 19:46:56,314 ERROR [main] > o.a.n.c.s.StandardControllerServiceProvider Failed to enable > JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536] > java.lang.IllegalStateException: Cannot invoke method public abstract > java.util.Set > org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields() > on Controller Service with identifier 0660348b-255e-1f3a--fb399783 > because the Controller Service is disabled > at > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84) > at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source) > at > org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42) > at > org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147) > at > org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149) > at > org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128) > at > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160) > at > org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126) > at > org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326) > at > org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441) > at > org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350) > at > org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152) > at > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042) > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312) > at > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554) > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) > at > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722) > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452) > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800) > at org.apache.nifi.NiFi.(NiFi.java:160) > at org.apache.nifi.NiFi.main(NiFi.java:267) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart
[ https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024053#comment-16024053 ] ASF GitHub Bot commented on NIFI-3972: -- Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/1855 initially it looked like it was sorted. But after another round of restarting all nodes in the cluster the problem occurred again. The case I have is a json record writer which depends on an external schema registry which could take a moment to enable and the writer is used by two kafka processors. The processors are not starting up as they should when the nodes startup because the writer service is disabled which is presumably because the service it depends on is not available. > Controller Service failing to enabled because the service it depends on is > not fully enabled on nifi restart > > > Key: NIFI-3972 > URL: https://issues.apache.org/jira/browse/NIFI-3972 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.3.0 > > > I have a JsonRecordSetWriter controller service. It depends on a Schema > Registry controller service. Both are enabled. Upon restart, the > JsonRecordSetWriter service fails to enable, indicating that the Schema > Registry service is not enabled. When I look at the Controller Service > configuration, though, the Schema Registry service is enabled. So it appears > that the JsonRecordSetWriter was attempting to be enabled before the Schema > Registry Service finishes enabling. > {code} > 2017-05-24 19:46:56,314 ERROR [main] > o.a.n.c.s.StandardControllerServiceProvider Failed to enable > JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536] > java.lang.IllegalStateException: Cannot invoke method public abstract > java.util.Set > org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields() > on Controller Service with identifier 0660348b-255e-1f3a--fb399783 > because the Controller Service is disabled > at > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84) > at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source) > at > org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42) > at > org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147) > at > org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149) > at > org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128) > at > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160) > at > org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126) > at > org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326) > at > org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441) > at > org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350) > at > org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152) > at > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042) > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312) > at > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554) > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) > at > org.apache.nifi.
[jira] [Updated] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart
[ https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-3972: -- Fix Version/s: 1.3.0 > Controller Service failing to enabled because the service it depends on is > not fully enabled on nifi restart > > > Key: NIFI-3972 > URL: https://issues.apache.org/jira/browse/NIFI-3972 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.3.0 > > > I have a JsonRecordSetWriter controller service. It depends on a Schema > Registry controller service. Both are enabled. Upon restart, the > JsonRecordSetWriter service fails to enable, indicating that the Schema > Registry service is not enabled. When I look at the Controller Service > configuration, though, the Schema Registry service is enabled. So it appears > that the JsonRecordSetWriter was attempting to be enabled before the Schema > Registry Service finishes enabling. > {code} > 2017-05-24 19:46:56,314 ERROR [main] > o.a.n.c.s.StandardControllerServiceProvider Failed to enable > JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536] > java.lang.IllegalStateException: Cannot invoke method public abstract > java.util.Set > org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields() > on Controller Service with identifier 0660348b-255e-1f3a--fb399783 > because the Controller Service is disabled > at > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84) > at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source) > at > org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42) > at > org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147) > at > org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149) > at > org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128) > at > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160) > at > org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126) > at > org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326) > at > org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441) > at > org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350) > at > org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152) > at > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042) > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312) > at > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554) > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) > at > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722) > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452) > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800) > at org.apache.nifi.NiFi.(NiFi.java:160) > at org.apache.nifi.NiFi.main(NiFi.java:267) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1855: NIFI-3972: Ensure that we wait until service state becomes...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/1855 initially it looked like it was sorted. But after another round of restarting all nodes in the cluster the problem occurred again. The case I have is a json record writer which depends on an external schema registry which could take a moment to enable and the writer is used by two kafka processors. The processors are not starting up as they should when the nodes startup because the writer service is disabled which is presumably because the service it depends on is not available. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024043#comment-16024043 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118404133 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleCsvFileLookupService.java --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.lookup; + +import java.io.FileNotFoundException; +import java.io.FileReader; +import java.io.IOException; +import java.nio.file.Paths; +import java.util.Arrays; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.locks.ReentrantLock; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import org.apache.commons.csv.CSVFormat; +import org.apache.commons.csv.CSVRecord; +import org.apache.commons.lang3.StringUtils; + +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnEnabled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.controller.AbstractControllerService; +import org.apache.nifi.controller.ControllerServiceInitializationContext; +import org.apache.nifi.controller.ConfigurationContext; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.reporting.InitializationException; +import org.apache.nifi.util.file.monitor.LastModifiedMonitor; +import org.apache.nifi.util.file.monitor.SynchronousFileWatcher; + +@Tags({"lookup", "cache", "enrich", "join", "csv", "reloadable", "key", "value"}) +@CapabilityDescription("A reloadable properties file-based lookup service") --- End diff -- "CSV file-based" instead of "properties". > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118404133 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleCsvFileLookupService.java --- @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.lookup; + +import java.io.FileNotFoundException; +import java.io.FileReader; +import java.io.IOException; +import java.nio.file.Paths; +import java.util.Arrays; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.locks.ReentrantLock; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import org.apache.commons.csv.CSVFormat; +import org.apache.commons.csv.CSVRecord; +import org.apache.commons.lang3.StringUtils; + +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnEnabled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.controller.AbstractControllerService; +import org.apache.nifi.controller.ControllerServiceInitializationContext; +import org.apache.nifi.controller.ConfigurationContext; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.reporting.InitializationException; +import org.apache.nifi.util.file.monitor.LastModifiedMonitor; +import org.apache.nifi.util.file.monitor.SynchronousFileWatcher; + +@Tags({"lookup", "cache", "enrich", "join", "csv", "reloadable", "key", "value"}) +@CapabilityDescription("A reloadable properties file-based lookup service") --- End diff -- "CSV file-based" instead of "properties". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024025#comment-16024025 ] ASF GitHub Bot commented on NIFI-3404: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1830 @jfrazee I understand, thank you! I am reviewing the updated commit now. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1830: NIFI-3404 Add lookup processor for enrichments/joins to re...
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/1830 @jfrazee I understand, thank you! I am reviewing the updated commit now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024021#comment-16024021 ] ASF GitHub Bot commented on NIFI-3404: -- Github user jfrazee commented on the issue: https://github.com/apache/nifi/pull/1830 @ijokarumawak The changes from [NIFI-3339](https://issues.apache.org/jira/browse/NIFI-3339)/#1450 have been removed from this PR along with the DatabaseLookupService. Will re-submit that stuff in another PR so we can move ahead with this and work out the details separately. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E > 2. https://github.com/jfrazee/nifi-lookup-service -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1830: NIFI-3404 Add lookup processor for enrichments/joins to re...
Github user jfrazee commented on the issue: https://github.com/apache/nifi/pull/1830 @ijokarumawak The changes from [NIFI-3339](https://issues.apache.org/jira/browse/NIFI-3339)/#1450 have been removed from this PR along with the DatabaseLookupService. Will re-submit that stuff in another PR so we can move ahead with this and work out the details separately. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning
[ https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024017#comment-16024017 ] Joseph Witt commented on NIFI-3971: --- under review > UpdateAttribute metric for output 'size' is larger than it should be when > cloning > - > > Key: NIFI-3971 > URL: https://issues.apache.org/jira/browse/NIFI-3971 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > > When I configure UpdateAttribute so that the 'success' relationship goes to > multiple destinations, the output count is double what it should be. For > instance, if i send in 1 GB of data, the output size should be 2 GB (since > it's cloned to a second connection) but it shows 4 GB. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3568) Default thread pool that is created for cluster request replication is not sufficient
[ https://issues.apache.org/jira/browse/NIFI-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-3568: -- Fix Version/s: 1.3.0 > Default thread pool that is created for cluster request replication is not > sufficient > - > > Key: NIFI-3568 > URL: https://issues.apache.org/jira/browse/NIFI-3568 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.3.0 > > > I have a cluster of 3 nodes. When the nodes are under heavy load, I notice > that API requests sometimes complete in 10's to 100's of milliseconds but > sometimes take several seconds (8+ seconds, at times). > After doing some investigation, it appears to be due to the fact that the > default thread pool size of 10 is not sufficient anymore. In the 0.x > baseline, it was okay because each time that a user clicks "Refresh" on the > UI it was a single request. With the 1.x baseline, this results in 4 separate > requests fired off simultaneously due to the multi-tenancy features added. As > a result, these 4 requests need to be replicated to 3 nodes each, which is 12 > web requests that have to occur. So even a simple Refresh on the UI cannot be > fully done in parallel. > Changing my pool size from 10 to 30 resulted in far more consistent response > times. Unfortunately, scaling the thread pool up to a large number of threads > can have its cons, too. So will create a "cached" thread pool and expose > properties for the "core pool size" and the "max pool size". -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3568) Default thread pool that is created for cluster request replication is not sufficient
[ https://issues.apache.org/jira/browse/NIFI-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-3568: -- Resolution: Fixed Status: Resolved (was: Patch Available) +1 merged to master. Cluster responsiveness was noticeably improved. > Default thread pool that is created for cluster request replication is not > sufficient > - > > Key: NIFI-3568 > URL: https://issues.apache.org/jira/browse/NIFI-3568 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > > I have a cluster of 3 nodes. When the nodes are under heavy load, I notice > that API requests sometimes complete in 10's to 100's of milliseconds but > sometimes take several seconds (8+ seconds, at times). > After doing some investigation, it appears to be due to the fact that the > default thread pool size of 10 is not sufficient anymore. In the 0.x > baseline, it was okay because each time that a user clicks "Refresh" on the > UI it was a single request. With the 1.x baseline, this results in 4 separate > requests fired off simultaneously due to the multi-tenancy features added. As > a result, these 4 requests need to be replicated to 3 nodes each, which is 12 > web requests that have to occur. So even a simple Refresh on the UI cannot be > fully done in parallel. > Changing my pool size from 10 to 30 resulted in far more consistent response > times. Unfortunately, scaling the thread pool up to a large number of threads > can have its cons, too. So will create a "cached" thread pool and expose > properties for the "core pool size" and the "max pool size". -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3568) Default thread pool that is created for cluster request replication is not sufficient
[ https://issues.apache.org/jira/browse/NIFI-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023993#comment-16023993 ] ASF subversion and git services commented on NIFI-3568: --- Commit 5aa3baca79eb6111c7b84e7adbdd7e1e50b71b8e in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5aa3bac ] NIFI-3568: This closes #1577. Use a cached thread pool in order to allow ThreadPoolRequestReplicator to scale up the number of threads to some configurable max Signed-off-by: joewitt > Default thread pool that is created for cluster request replication is not > sufficient > - > > Key: NIFI-3568 > URL: https://issues.apache.org/jira/browse/NIFI-3568 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > > I have a cluster of 3 nodes. When the nodes are under heavy load, I notice > that API requests sometimes complete in 10's to 100's of milliseconds but > sometimes take several seconds (8+ seconds, at times). > After doing some investigation, it appears to be due to the fact that the > default thread pool size of 10 is not sufficient anymore. In the 0.x > baseline, it was okay because each time that a user clicks "Refresh" on the > UI it was a single request. With the 1.x baseline, this results in 4 separate > requests fired off simultaneously due to the multi-tenancy features added. As > a result, these 4 requests need to be replicated to 3 nodes each, which is 12 > web requests that have to occur. So even a simple Refresh on the UI cannot be > fully done in parallel. > Changing my pool size from 10 to 30 resulted in far more consistent response > times. Unfortunately, scaling the thread pool up to a large number of threads > can have its cons, too. So will create a "cached" thread pool and expose > properties for the "core pool size" and the "max pool size". -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1577: NIFI-3568: Use a cached thread pool in order to all...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1577 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3568) Default thread pool that is created for cluster request replication is not sufficient
[ https://issues.apache.org/jira/browse/NIFI-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023994#comment-16023994 ] ASF GitHub Bot commented on NIFI-3568: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1577 > Default thread pool that is created for cluster request replication is not > sufficient > - > > Key: NIFI-3568 > URL: https://issues.apache.org/jira/browse/NIFI-3568 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > > I have a cluster of 3 nodes. When the nodes are under heavy load, I notice > that API requests sometimes complete in 10's to 100's of milliseconds but > sometimes take several seconds (8+ seconds, at times). > After doing some investigation, it appears to be due to the fact that the > default thread pool size of 10 is not sufficient anymore. In the 0.x > baseline, it was okay because each time that a user clicks "Refresh" on the > UI it was a single request. With the 1.x baseline, this results in 4 separate > requests fired off simultaneously due to the multi-tenancy features added. As > a result, these 4 requests need to be replicated to 3 nodes each, which is 12 > web requests that have to occur. So even a simple Refresh on the UI cannot be > fully done in parallel. > Changing my pool size from 10 to 30 resulted in far more consistent response > times. Unfortunately, scaling the thread pool up to a large number of threads > can have its cons, too. So will create a "cached" thread pool and expose > properties for the "core pool size" and the "max pool size". -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart
[ https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023990#comment-16023990 ] Joseph Witt commented on NIFI-3972: --- have this in review. Looking much better. > Controller Service failing to enabled because the service it depends on is > not fully enabled on nifi restart > > > Key: NIFI-3972 > URL: https://issues.apache.org/jira/browse/NIFI-3972 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > > I have a JsonRecordSetWriter controller service. It depends on a Schema > Registry controller service. Both are enabled. Upon restart, the > JsonRecordSetWriter service fails to enable, indicating that the Schema > Registry service is not enabled. When I look at the Controller Service > configuration, though, the Schema Registry service is enabled. So it appears > that the JsonRecordSetWriter was attempting to be enabled before the Schema > Registry Service finishes enabling. > {code} > 2017-05-24 19:46:56,314 ERROR [main] > o.a.n.c.s.StandardControllerServiceProvider Failed to enable > JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536] > java.lang.IllegalStateException: Cannot invoke method public abstract > java.util.Set > org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields() > on Controller Service with identifier 0660348b-255e-1f3a--fb399783 > because the Controller Service is disabled > at > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84) > at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source) > at > org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42) > at > org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147) > at > org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149) > at > org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128) > at > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160) > at > org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126) > at > org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326) > at > org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441) > at > org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350) > at > org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152) > at > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042) > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312) > at > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554) > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) > at > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722) > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452) > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800) > at org.apache.nifi.NiFi.(NiFi.java:160) > at org.apache.nifi.NiFi.main(NiFi.java:267) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (NIFI-3973) Create a new Kudu Processor to ingest data
[ https://issues.apache.org/jira/browse/NIFI-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne reassigned NIFI-3973: Assignee: Cam Quoc Mach > Create a new Kudu Processor to ingest data > -- > > Key: NIFI-3973 > URL: https://issues.apache.org/jira/browse/NIFI-3973 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Cam Mach >Assignee: Cam Quoc Mach > Original Estimate: 336h > Remaining Estimate: 336h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3973) Create a new Kudu Processor to ingest data
[ https://issues.apache.org/jira/browse/NIFI-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023958#comment-16023958 ] Cam Mach commented on NIFI-3973: Hi, I would like to contribute my Kudu processor to the project. Please review and let me know what need to be adjusted. > Create a new Kudu Processor to ingest data > -- > > Key: NIFI-3973 > URL: https://issues.apache.org/jira/browse/NIFI-3973 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Cam Mach > Original Estimate: 336h > Remaining Estimate: 336h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3973) Create a new Kudu Processor to ingest data
[ https://issues.apache.org/jira/browse/NIFI-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cam Mach updated NIFI-3973: --- Summary: Create a new Kudu Processor to ingest data (was: Create a new Kudu Process to ingest data) > Create a new Kudu Processor to ingest data > -- > > Key: NIFI-3973 > URL: https://issues.apache.org/jira/browse/NIFI-3973 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Cam Mach > Original Estimate: 336h > Remaining Estimate: 336h > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3973) Create a new Kudu Process to ingest data
Cam Mach created NIFI-3973: -- Summary: Create a new Kudu Process to ingest data Key: NIFI-3973 URL: https://issues.apache.org/jira/browse/NIFI-3973 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Cam Mach -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi-minifi-cpp issue #104: MINIFI-296 - Configurable logging, spdlog 0.13.0
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/104 @brosander yeah I think an atomic would solve the problem. yeah, I think the pattern would be cool. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #104: MINIFI-296 - Configurable logging, spdlog 0.13.0
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/104 @phrocker For your first point about reinitializing logging while a flow is running, is it worth it to switch the delegate reference on the Logger to an atomic? Second point I think should be fairly easily addressable with a format pattern in the logger config. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (NIFI-3925) Add Template list with long template name
[ https://issues.apache.org/jira/browse/NIFI-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-3925: -- Resolution: Fixed Fix Version/s: 1.3.0 Status: Resolved (was: Patch Available) > Add Template list with long template name > - > > Key: NIFI-3925 > URL: https://issues.apache.org/jira/browse/NIFI-3925 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.2.0 >Reporter: Mark Bean >Assignee: Scott Aslan >Priority: Minor > Fix For: 1.3.0 > > > If a template has a very long name (roughly 40 characters or more), the > dropdown list of templates in "Add Template" does not display properly. The > "?" rollover which displays the template description is shown on the next > line. In effect, the long-named template line has no "?" rollover, and the > next template line has two adjacent "?" rollovers (one corresponding to the > previous, long-named template and one corresponding to itself.) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3925) Add Template list with long template name
[ https://issues.apache.org/jira/browse/NIFI-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023617#comment-16023617 ] ASF GitHub Bot commented on NIFI-3925: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/1848 Thanks @scottyaslan. This has been merged to master. > Add Template list with long template name > - > > Key: NIFI-3925 > URL: https://issues.apache.org/jira/browse/NIFI-3925 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.2.0 >Reporter: Mark Bean >Assignee: Scott Aslan >Priority: Minor > > If a template has a very long name (roughly 40 characters or more), the > dropdown list of templates in "Add Template" does not display properly. The > "?" rollover which displays the template description is shown on the next > line. In effect, the long-named template line has no "?" rollover, and the > next template line has two adjacent "?" rollovers (one corresponding to the > previous, long-named template and one corresponding to itself.) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1848: [NIFI-3925] bound width of combo options drop down text
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/1848 Thanks @scottyaslan. This has been merged to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3925) Add Template list with long template name
[ https://issues.apache.org/jira/browse/NIFI-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023612#comment-16023612 ] ASF GitHub Bot commented on NIFI-3925: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1848 > Add Template list with long template name > - > > Key: NIFI-3925 > URL: https://issues.apache.org/jira/browse/NIFI-3925 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.2.0 >Reporter: Mark Bean >Assignee: Scott Aslan >Priority: Minor > > If a template has a very long name (roughly 40 characters or more), the > dropdown list of templates in "Add Template" does not display properly. The > "?" rollover which displays the template description is shown on the next > line. In effect, the long-named template line has no "?" rollover, and the > next template line has two adjacent "?" rollovers (one corresponding to the > previous, long-named template and one corresponding to itself.) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3925) Add Template list with long template name
[ https://issues.apache.org/jira/browse/NIFI-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023609#comment-16023609 ] ASF subversion and git services commented on NIFI-3925: --- Commit 86728bac7ef091dba79cd3ffd3219c80f4fca8ba in nifi's branch refs/heads/master from [~scottyaslan] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=86728ba ] [NIFI-3925] for the jquery.combo plugin: update the calculation of the width of the combo box options overlay and also updated the plugins styles to now leverage the css calc() to determine the exact width of .combotext and .combo-option-text elements. This closes #1848 > Add Template list with long template name > - > > Key: NIFI-3925 > URL: https://issues.apache.org/jira/browse/NIFI-3925 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.2.0 >Reporter: Mark Bean >Assignee: Scott Aslan >Priority: Minor > > If a template has a very long name (roughly 40 characters or more), the > dropdown list of templates in "Add Template" does not display properly. The > "?" rollover which displays the template description is shown on the next > line. In effect, the long-named template line has no "?" rollover, and the > next template line has two adjacent "?" rollovers (one corresponding to the > previous, long-named template and one corresponding to itself.) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1848: [NIFI-3925] bound width of combo options drop down ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1848 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #104: MINIFI-296 - Configurable logging, spdlog 0.13.0
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/104 This is good stuff. I'm taking a look General comments before I dive in. Did we lose the ability to change the logger format real time? If there was an issue in production and we had to change the logger to a null appender, restarting may not always be possible, whereas sending a shared object through a p2p c2 protocol would allow us to dynamically load and change loggers without impacting the implementation, for example ( kind of a contrived example) Additionally, I Noticed that my logs rolled much more frequently because we have an additional 52 bytes per line in some cases. Is there a way to limit that? It slowed my runs down because there was much more memory allocated by spdlog log statements when threading was high and I tested with trace. Even to the class name would be useful. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart
[ https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-3972: - Status: Patch Available (was: Open) > Controller Service failing to enabled because the service it depends on is > not fully enabled on nifi restart > > > Key: NIFI-3972 > URL: https://issues.apache.org/jira/browse/NIFI-3972 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > > I have a JsonRecordSetWriter controller service. It depends on a Schema > Registry controller service. Both are enabled. Upon restart, the > JsonRecordSetWriter service fails to enable, indicating that the Schema > Registry service is not enabled. When I look at the Controller Service > configuration, though, the Schema Registry service is enabled. So it appears > that the JsonRecordSetWriter was attempting to be enabled before the Schema > Registry Service finishes enabling. > {code} > 2017-05-24 19:46:56,314 ERROR [main] > o.a.n.c.s.StandardControllerServiceProvider Failed to enable > JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536] > java.lang.IllegalStateException: Cannot invoke method public abstract > java.util.Set > org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields() > on Controller Service with identifier 0660348b-255e-1f3a--fb399783 > because the Controller Service is disabled > at > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84) > at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source) > at > org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42) > at > org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147) > at > org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149) > at > org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128) > at > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160) > at > org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126) > at > org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326) > at > org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441) > at > org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384) > at > org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350) > at > org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159) > at > org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152) > at > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042) > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312) > at > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554) > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) > at > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722) > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452) > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800) > at org.apache.nifi.NiFi.(NiFi.java:160) > at org.apache.nifi.NiFi.main(NiFi.java:267) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart
[ https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023606#comment-16023606 ] ASF GitHub Bot commented on NIFI-3972: -- GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/1855 NIFI-3972: Ensure that we wait until service state becomes enabled be… …fore triggering completable future that says that it's enabled Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-3972 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1855.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1855 commit 27ca0d809e3105993396d80b7efb5268f68a263d Author: Mark Payne Date: 2017-05-24T20:15:18Z NIFI-3972: Ensure that we wait until service state becomes enabled before triggering completable future that says that it's enabled > Controller Service failing to enabled because the service it depends on is > not fully enabled on nifi restart > > > Key: NIFI-3972 > URL: https://issues.apache.org/jira/browse/NIFI-3972 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > > I have a JsonRecordSetWriter controller service. It depends on a Schema > Registry controller service. Both are enabled. Upon restart, the > JsonRecordSetWriter service fails to enable, indicating that the Schema > Registry service is not enabled. When I look at the Controller Service > configuration, though, the Schema Registry service is enabled. So it appears > that the JsonRecordSetWriter was attempting to be enabled before the Schema > Registry Service finishes enabling. > {code} > 2017-05-24 19:46:56,314 ERROR [main] > o.a.n.c.s.StandardControllerServiceProvider Failed to enable > JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536] > java.lang.IllegalStateException: Cannot invoke method public abstract > java.util.Set > org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields() > on Controller Service with identifier 0660348b-255e-1f3a--fb399783 > because the Controller Service is disabled > at > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84) > at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source) > at > org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42) > at > org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147) > at > org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149) > at > org.apache.nif
[GitHub] nifi pull request #1855: NIFI-3972: Ensure that we wait until service state ...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/1855 NIFI-3972: Ensure that we wait until service state becomes enabled be⦠â¦fore triggering completable future that says that it's enabled Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-3972 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1855.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1855 commit 27ca0d809e3105993396d80b7efb5268f68a263d Author: Mark Payne Date: 2017-05-24T20:15:18Z NIFI-3972: Ensure that we wait until service state becomes enabled before triggering completable future that says that it's enabled --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart
Mark Payne created NIFI-3972: Summary: Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart Key: NIFI-3972 URL: https://issues.apache.org/jira/browse/NIFI-3972 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne I have a JsonRecordSetWriter controller service. It depends on a Schema Registry controller service. Both are enabled. Upon restart, the JsonRecordSetWriter service fails to enable, indicating that the Schema Registry service is not enabled. When I look at the Controller Service configuration, though, the Schema Registry service is enabled. So it appears that the JsonRecordSetWriter was attempting to be enabled before the Schema Registry Service finishes enabling. {code} 2017-05-24 19:46:56,314 ERROR [main] o.a.n.c.s.StandardControllerServiceProvider Failed to enable JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536] java.lang.IllegalStateException: Cannot invoke method public abstract java.util.Set org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields() on Controller Service with identifier 0660348b-255e-1f3a--fb399783 because the Controller Service is disabled at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84) at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source) at org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42) at org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147) at org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149) at org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128) at org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160) at org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126) at org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326) at org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441) at org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301) at org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327) at org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384) at org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350) at org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271) at org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159) at org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152) at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042) at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554) at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800) at org.apache.nifi.NiFi.(NiFi.java:160) at org.apache.nifi.NiFi.main(NiFi.java:267) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled
[ https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-3963: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Remote Process Group does not honor yield duration if transmission is > disabled then reenabled > - > > Key: NIFI-3963 > URL: https://issues.apache.org/jira/browse/NIFI-3963 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Joseph Percivall >Assignee: Matt Gilman > Fix For: 1.3.0 > > Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, > unable-to-reproduce-nifi-3963.png > > > To reproduce, create two NiFi instances properly set up for unsecure or > secure S2S. One on side add an input/output port and the other an RPG. > Enable transmission on the RPG. Stop the instance with the port. On the > running instance see NiFi honoring the yield duration. Then stop and start > transmission, see NiFi repeatedly attempt to connect to the instance (I saw > multiple log messages per millisecond). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled
[ https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023566#comment-16023566 ] ASF subversion and git services commented on NIFI-3963: --- Commit f97b3fe455e7cf8bd051c7a40f3e3f7ee1d1a16a in nifi's branch refs/heads/master from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=f97b3fe ] NIFI-3963: - Ensuring the RemoteGroupPort yields when the details cannot be refreshed from any of the configured remote instances. This closes #1853. Signed-off-by: Bryan Bende > Remote Process Group does not honor yield duration if transmission is > disabled then reenabled > - > > Key: NIFI-3963 > URL: https://issues.apache.org/jira/browse/NIFI-3963 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Joseph Percivall >Assignee: Matt Gilman > Fix For: 1.3.0 > > Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, > unable-to-reproduce-nifi-3963.png > > > To reproduce, create two NiFi instances properly set up for unsecure or > secure S2S. One on side add an input/output port and the other an RPG. > Enable transmission on the RPG. Stop the instance with the port. On the > running instance see NiFi honoring the yield duration. Then stop and start > transmission, see NiFi repeatedly attempt to connect to the instance (I saw > multiple log messages per millisecond). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled
[ https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023568#comment-16023568 ] ASF GitHub Bot commented on NIFI-3963: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1853 > Remote Process Group does not honor yield duration if transmission is > disabled then reenabled > - > > Key: NIFI-3963 > URL: https://issues.apache.org/jira/browse/NIFI-3963 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Joseph Percivall >Assignee: Matt Gilman > Fix For: 1.3.0 > > Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, > unable-to-reproduce-nifi-3963.png > > > To reproduce, create two NiFi instances properly set up for unsecure or > secure S2S. One on side add an input/output port and the other an RPG. > Enable transmission on the RPG. Stop the instance with the port. On the > running instance see NiFi honoring the yield duration. Then stop and start > transmission, see NiFi repeatedly attempt to connect to the instance (I saw > multiple log messages per millisecond). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1853: NIFI-3963: RPG Yield Issue
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1853 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled
[ https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023563#comment-16023563 ] ASF GitHub Bot commented on NIFI-3963: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/1853 +1 Looks good, repeated the steps outline by Joe P and can see that we are now yielding correctly, will merge to master > Remote Process Group does not honor yield duration if transmission is > disabled then reenabled > - > > Key: NIFI-3963 > URL: https://issues.apache.org/jira/browse/NIFI-3963 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Joseph Percivall >Assignee: Matt Gilman > Fix For: 1.3.0 > > Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, > unable-to-reproduce-nifi-3963.png > > > To reproduce, create two NiFi instances properly set up for unsecure or > secure S2S. One on side add an input/output port and the other an RPG. > Enable transmission on the RPG. Stop the instance with the port. On the > running instance see NiFi honoring the yield duration. Then stop and start > transmission, see NiFi repeatedly attempt to connect to the instance (I saw > multiple log messages per millisecond). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1853: NIFI-3963: RPG Yield Issue
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/1853 +1 Looks good, repeated the steps outline by Joe P and can see that we are now yielding correctly, will merge to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning
[ https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023511#comment-16023511 ] ASF GitHub Bot commented on NIFI-3971: -- GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/1854 NIFI-3971: Fixed bug in calculating content size that was transferred… … when cloning a relationship Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-3971 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1854.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1854 commit 8d4bc8a2301535c6ca6041bda3c0ab35ee7531ea Author: Mark Payne Date: 2017-05-24T19:31:33Z NIFI-3971: Fixed bug in calculating content size that was transferred when cloning a relationship > UpdateAttribute metric for output 'size' is larger than it should be when > cloning > - > > Key: NIFI-3971 > URL: https://issues.apache.org/jira/browse/NIFI-3971 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > > When I configure UpdateAttribute so that the 'success' relationship goes to > multiple destinations, the output count is double what it should be. For > instance, if i send in 1 GB of data, the output size should be 2 GB (since > it's cloned to a second connection) but it shows 4 GB. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi pull request #1854: NIFI-3971: Fixed bug in calculating content size th...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/1854 NIFI-3971: Fixed bug in calculating content size that was transferred⦠⦠when cloning a relationship Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-3971 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1854.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1854 commit 8d4bc8a2301535c6ca6041bda3c0ab35ee7531ea Author: Mark Payne Date: 2017-05-24T19:31:33Z NIFI-3971: Fixed bug in calculating content size that was transferred when cloning a relationship --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (NIFI-3970) Add CSVRecordLookupService
Joey Frazee created NIFI-3970: - Summary: Add CSVRecordLookupService Key: NIFI-3970 URL: https://issues.apache.org/jira/browse/NIFI-3970 Project: Apache NiFi Issue Type: Bug Reporter: Joey Frazee Assignee: Joey Frazee PR [#1830|https://github.com/apache/nifi/pull/1830] provides a SimpleCsvFileLookupService. Since CSV data is tabular, a counterpart CSVRecordLookupService would be useful for using with multi-criteria lookups and enrichments in LookupRecord and LookupAttribute. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning
Mark Payne created NIFI-3971: Summary: UpdateAttribute metric for output 'size' is larger than it should be when cloning Key: NIFI-3971 URL: https://issues.apache.org/jira/browse/NIFI-3971 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne When I configure UpdateAttribute so that the 'success' relationship goes to multiple destinations, the output count is double what it should be. For instance, if i send in 1 GB of data, the output size should be 2 GB (since it's cloned to a second connection) but it shows 4 GB. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi-minifi-cpp pull request #104: MINIFI-296 - Configurable logging, spdlog...
GitHub user brosander opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/104 MINIFI-296 - Configurable logging, spdlog 0.13.0 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file? - [X] - N/A - If applicable, have you updated the NOTICE file? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi-minifi-cpp MINIFI-296 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/104.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #104 commit fa0304cd48920a1d3b259b9a51a6aff0c79a88e7 Author: Bryan Rosander Date: 2017-05-11T19:42:54Z MINIFI-296 - Configurable logging, spdlog 0.13.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled
[ https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023483#comment-16023483 ] ASF GitHub Bot commented on NIFI-3963: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/1853 Reviewing... > Remote Process Group does not honor yield duration if transmission is > disabled then reenabled > - > > Key: NIFI-3963 > URL: https://issues.apache.org/jira/browse/NIFI-3963 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Joseph Percivall >Assignee: Matt Gilman > Fix For: 1.3.0 > > Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, > unable-to-reproduce-nifi-3963.png > > > To reproduce, create two NiFi instances properly set up for unsecure or > secure S2S. One on side add an input/output port and the other an RPG. > Enable transmission on the RPG. Stop the instance with the port. On the > running instance see NiFi honoring the yield duration. Then stop and start > transmission, see NiFi repeatedly attempt to connect to the instance (I saw > multiple log messages per millisecond). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] nifi issue #1853: NIFI-3963: RPG Yield Issue
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/1853 Reviewing... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data
[ https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023479#comment-16023479 ] ASF GitHub Bot commented on NIFI-3404: -- Github user jfrazee commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118341314 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java --- @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +import org.apache.commons.lang3.StringUtils; + +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.lookup.LookupFailureException; +import org.apache.nifi.lookup.LookupService; +import org.apache.nifi.lookup.StringLookupService; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + +@EventDriven +@SideEffectFree +@SupportsBatching +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes", "Attribute Expression Language"}) --- End diff -- Holdover from the interfaces this was originally written against. There was a LookupService and a MutableLookupService. Will remove. > Add lookup processor for enrichments/joins to reference data > > > Key: NIFI-3404 > URL: https://issues.apache.org/jira/browse/NIFI-3404 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joey Frazee >Assignee: Joey Frazee > > NiFi doesn't currently have an easy, concise way of doing enrichment, joining > against reference data sets or performing attribute lookups against external > data sources. > Since enrichments and joins are basic streaming use cases, and since > attributes and EL are often used to parameterize processor properties, there > is a need for an easy way to do enrichments, joins and lookups without having > to write code or create a lengthy data flow. > There's been some discussion of this on the mailing list [1] and I've started > work on a LookupAttribute [2] processor that delegates the work to controller > services. > 1. > https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0
[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...
Github user jfrazee commented on a diff in the pull request: https://github.com/apache/nifi/pull/1830#discussion_r118341314 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java --- @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +import org.apache.commons.lang3.StringUtils; + +import org.apache.nifi.annotation.behavior.DynamicProperty; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.lookup.LookupFailureException; +import org.apache.nifi.lookup.LookupService; +import org.apache.nifi.lookup.StringLookupService; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; + +@EventDriven +@SideEffectFree +@SupportsBatching +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes", "Attribute Expression Language"}) --- End diff -- Holdover from the interfaces this was originally written against. There was a LookupService and a MutableLookupService. Will remove. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3568) Default thread pool that is created for cluster request replication is not sufficient
[ https://issues.apache.org/jira/browse/NIFI-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023463#comment-16023463 ] Mark Payne commented on NIFI-3568: -- [~joewitt] - rebased. Thanks! > Default thread pool that is created for cluster request replication is not > sufficient > - > > Key: NIFI-3568 > URL: https://issues.apache.org/jira/browse/NIFI-3568 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > > I have a cluster of 3 nodes. When the nodes are under heavy load, I notice > that API requests sometimes complete in 10's to 100's of milliseconds but > sometimes take several seconds (8+ seconds, at times). > After doing some investigation, it appears to be due to the fact that the > default thread pool size of 10 is not sufficient anymore. In the 0.x > baseline, it was okay because each time that a user clicks "Refresh" on the > UI it was a single request. With the 1.x baseline, this results in 4 separate > requests fired off simultaneously due to the multi-tenancy features added. As > a result, these 4 requests need to be replicated to 3 nodes each, which is 12 > web requests that have to occur. So even a simple Refresh on the UI cannot be > fully done in parallel. > Changing my pool size from 10 to 30 resulted in far more consistent response > times. Unfortunately, scaling the thread pool up to a large number of threads > can have its cons, too. So will create a "cached" thread pool and expose > properties for the "core pool size" and the "max pool size". -- This message was sent by Atlassian JIRA (v6.3.15#6346)