[jira] [Commented] (NIFI-3484) GenerateTableFetch Should Allow for Right Boundary

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024236#comment-16024236
 ] 

ASF GitHub Bot commented on NIFI-3484:
--

Github user patricker commented on the issue:

https://github.com/apache/nifi/pull/1513
  
@ilyatau Why do you feel that this request isn't finished?
While I do have a newer version, the functionality it adds is a bit more 
advanced and I've had a hard time automating the testing.

The additional functionality I have working will generate a premature right 
boundary to reduce the number of records brought back. You might think that 
this is already the point of Generate Table Fetch, but I found that even with 
an indexed timestamp column you still have to page the index when paging 
through data on some systems. In one case we found that on SAP Hana we ran into 
some internal limitations which did not allow you to page past more than 2 
billion rows of data. The table we were loading with Generate Table Fetch had 
~6 billion rows. Using the un-committed change you can provide a per execution 
limit to Generate Table Fetch so that it will only generate pages of `x` size 
for the first `y` rows in the table.

When I tried to test this on SQL Lite it did not work, though it is working 
on other SQL systems we've tried it on.


> GenerateTableFetch Should Allow for Right Boundary
> --
>
> Key: NIFI-3484
> URL: https://issues.apache.org/jira/browse/NIFI-3484
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Peter Wicks
>Assignee: Peter Wicks
>Priority: Minor
>
> When using GenerateTableFetch it places no right hand boundary on pages of 
> data.  This can lead to issues when the statement says to get the next 1000 
> records greater then a specific key, but records were added to the table 
> between the time the processor executed and when the SQL is being executed. 
> As a result it pulls in records that did not exist when the processor was 
> run.  On the next execution of the processor these records will be pulled in 
> a second time.
> Example:
> Partition Size = 1000
> First run (no state): Count(*)=4700 and MAX(ID)=4700.
> 5 FlowFiles are generated, the last one will say to fetch 1000, not 700. (But 
> I don't think this is really a bug, just an observation).
> 5 Flow Files are now in queue to be executed by ExecuteSQL.  Before the 5th 
> file can execute 400 new rows are added to the table.  When the final SQL 
> statement is executed 300 extra records, with higher ID values, will also be 
> pulled into NiFi.
> Second run (state: ID=4700).  Count(*) ID>4700 = 400 and MAX(ID)=5100.
> 1 Flow File is generated, but includes 300 records already pulled into NiFI.
> The solution is to have an optional property that will let users use the new 
> MAX(ID) as a right boundary when generating queries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1513: NIFI-3484 GenerateTableFetch Should Allow for Right Bounda...

[patricker]

Github user patricker commented on the issue:

https://github.com/apache/nifi/pull/1513
  
@ilyatau Why do you feel that this request isn't finished?
While I do have a newer version, the functionality it adds is a bit more 
advanced and I've had a hard time automating the testing.

The additional functionality I have working will generate a premature right 
boundary to reduce the number of records brought back. You might think that 
this is already the point of Generate Table Fetch, but I found that even with 
an indexed timestamp column you still have to page the index when paging 
through data on some systems. In one case we found that on SAP Hana we ran into 
some internal limitations which did not allow you to page past more than 2 
billion rows of data. The table we were loading with Generate Table Fetch had 
~6 billion rows. Using the un-committed change you can provide a per execution 
limit to Generate Table Fetch so that it will only generate pages of `x` size 
for the first `y` rows in the table.

When I tried to test this on SQL Lite it did not work, though it is working 
on other SQL systems we've tried it on.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1777: NIFI-3859 - Provide filtering options in S2SProvenanceRepo...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/1777
  
Thanks @ijokarumawak !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3859) Provide filtering options in S2SProvenanceReportingTask

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-3859:

   Resolution: Fixed
Fix Version/s: 1.3.0
   Status: Resolved  (was: Patch Available)

> Provide filtering options in S2SProvenanceReportingTask
> ---
>
> Key: NIFI-3859
> URL: https://issues.apache.org/jira/browse/NIFI-3859
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Minor
> Fix For: 1.3.0
>
>
> Based on what will be used the S2SProvenanceReportingTask for, it could be 
> nice to provide some filtering options to the user in order to generate flow 
> files only for a subset of the last provenance events.
> Filters could be:
> - Event type
> - Component type
> - Component ID



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3859) Provide filtering options in S2SProvenanceReportingTask

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024204#comment-16024204
 ] 

ASF subversion and git services commented on NIFI-3859:
---

Commit b6eb0ac0fb382b5cc2e3f60196e36835437883f3 in nifi's branch 
refs/heads/master from [~pvillard]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b6eb0ac ]

NIFI-3859 - Provide filtering options in S2SProvenanceReportingTask

This closes #1777.

Signed-off-by: Koji Kawamura 


> Provide filtering options in S2SProvenanceReportingTask
> ---
>
> Key: NIFI-3859
> URL: https://issues.apache.org/jira/browse/NIFI-3859
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Minor
>
> Based on what will be used the S2SProvenanceReportingTask for, it could be 
> nice to provide some filtering options to the user in order to generate flow 
> files only for a subset of the last provenance events.
> Filters could be:
> - Event type
> - Component type
> - Component ID



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3859) Provide filtering options in S2SProvenanceReportingTask

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024205#comment-16024205
 ] 

ASF GitHub Bot commented on NIFI-3859:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1777


> Provide filtering options in S2SProvenanceReportingTask
> ---
>
> Key: NIFI-3859
> URL: https://issues.apache.org/jira/browse/NIFI-3859
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Minor
>
> Based on what will be used the S2SProvenanceReportingTask for, it could be 
> nice to provide some filtering options to the user in order to generate flow 
> files only for a subset of the last provenance events.
> Filters could be:
> - Event type
> - Component type
> - Component ID



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1777: NIFI-3859 - Provide filtering options in S2SProvena...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1777


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1777: NIFI-3859 - Provide filtering options in S2SProvena...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1777#discussion_r118419716
  
--- Diff: 
nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java
 ---
@@ -67,14 +73,73 @@
 
 static final PropertyDescriptor PLATFORM = new 
PropertyDescriptor.Builder()
 .name("Platform")
+.displayName("Platform")
 .description("The value to use for the platform field in each 
provenance event.")
 .required(true)
 .expressionLanguageSupported(true)
 .defaultValue("nifi")
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
 .build();
 
+static final PropertyDescriptor FILTER_EVENT_TYPE = new 
PropertyDescriptor.Builder()
--- End diff --

These newly added properties are not added to 
`getSupportedPropertyDescriptors` method. So I was not able to configure 
filters from NIFI UI.

Since this is a simple fix, I went ahead and added these filters and able 
to confirm provenance events are filtered. LGTM, +1. I will merge this into 
master. Thanks @pvillard31 !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3859) Provide filtering options in S2SProvenanceReportingTask

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024203#comment-16024203
 ] 

ASF GitHub Bot commented on NIFI-3859:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1777#discussion_r118419716
  
--- Diff: 
nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java
 ---
@@ -67,14 +73,73 @@
 
 static final PropertyDescriptor PLATFORM = new 
PropertyDescriptor.Builder()
 .name("Platform")
+.displayName("Platform")
 .description("The value to use for the platform field in each 
provenance event.")
 .required(true)
 .expressionLanguageSupported(true)
 .defaultValue("nifi")
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
 .build();
 
+static final PropertyDescriptor FILTER_EVENT_TYPE = new 
PropertyDescriptor.Builder()
--- End diff --

These newly added properties are not added to 
`getSupportedPropertyDescriptors` method. So I was not able to configure 
filters from NIFI UI.

Since this is a simple fix, I went ahead and added these filters and able 
to confirm provenance events are filtered. LGTM, +1. I will merge this into 
master. Thanks @pvillard31 !


> Provide filtering options in S2SProvenanceReportingTask
> ---
>
> Key: NIFI-3859
> URL: https://issues.apache.org/jira/browse/NIFI-3859
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Minor
>
> Based on what will be used the S2SProvenanceReportingTask for, it could be 
> nice to provide some filtering options to the user in order to generate flow 
> files only for a subset of the last provenance events.
> Filters could be:
> - Event type
> - Component type
> - Component ID



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-1709) The nifi.sh install script assumes RHEL directories

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024169#comment-16024169
 ] 

ASF GitHub Bot commented on NIFI-1709:
--

Github user jfrazee commented on the issue:

https://github.com/apache/nifi/pull/1794
  
@trixpan Fair enough. Seems clear the lsb_release suggestion was bad. BTW, 
I tested your changes on opensuse:latest (leap) and it's working as expected.


> The nifi.sh install script assumes RHEL directories
> ---
>
> Key: NIFI-1709
> URL: https://issues.apache.org/jira/browse/NIFI-1709
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
> Environment: SUSE
>Reporter: David A. Wynne
>Assignee: Andre F de Miranda
>Priority: Minor
>
> When setting up NiFi, the command: 
> bin/nifi.sh install
> The following error occurs:
> ln: failed to create symbolic link `/etc/rc2.d/S65nifi': No such file or 
> directory
> ln: failed to create symbolic link `/etc/rc2.d/K65nifi': No such file or 
> directory
> Service nifi installed
> Looking in the  nifi.sh, around line 145 - 148, you see: 
> rm -f "/etc/rc2.d/S65${SVC_NAME}"
> ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/S65${SVC_NAME}"
> rm -f "/etc/rc2.d/K65${SVC_NAME}"
> ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}"
> It tries to symlink from /etc/init.d/nifi to /etc/rc2.d/S65nifi  (and 
> K65nifi). 
> The problem is that the script assumes that /etc/rc2.d exists in a SUSE 
> system, which it doesn't. 
> In Suse11, this directory is /etc/init.d/rc2.d/
> The script, especially the "install" option should correctly identify the OS 
> flavor it runs on, and install the files in the correct location. 
> The next version of RHEL (8, not out yet), and Ubuntu 16.02 LTS (releasing 
> next month), will use systemD, which does not have the concept of /etc/rc2.d 
> directories, but works with "targets". The script will not be compatible with 
> these OS changes. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1794: NIFI-1709 - Introduce logic to probe Linux version using /...

[jfrazee]

Github user jfrazee commented on the issue:

https://github.com/apache/nifi/pull/1794
  
@trixpan Fair enough. Seems clear the lsb_release suggestion was bad. BTW, 
I tested your changes on opensuse:latest (leap) and it's working as expected.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3640) GetEventHub should not assume domain names

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024119#comment-16024119
 ] 

ASF GitHub Bot commented on NIFI-3640:
--

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/1617
  
Reviewing...


> GetEventHub should not assume domain names
> --
>
> Key: NIFI-3640
> URL: https://issues.apache.org/jira/browse/NIFI-3640
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Joseph Niemiec
>Assignee: Joseph Niemiec
>
> Today the GetEventHubProcessor hard codes in the servicebus host from Azure 
> as ".servicebus.windows.net"  this does not allow us to deploy to other Geo's 
> where the URL changes such as "servicebus.chinacloudapi.cn"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1617: NIFI-3640 uri eventhub changes

[jtstorck]

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/1617
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...

[jfrazee]

Github user jfrazee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1830#discussion_r118408976
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java
 ---
@@ -47,6 +47,7 @@ protected PropertyDescriptor 
getSupportedDynamicPropertyDescriptor(final String
 .required(false)
 .dynamic(true)
 .addValidator(Validator.VALID)
+.expressionLanguageSupported(true)
--- End diff --

I meant to remove the expressionLanguageSupported(true). I don't think I 
had an especially good reason to have added the EL support to the dynamic 
property for the CS in the first place.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024105#comment-16024105
 ] 

ASF GitHub Bot commented on NIFI-3404:
--

Github user jfrazee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1830#discussion_r118408976
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java
 ---
@@ -47,6 +47,7 @@ protected PropertyDescriptor 
getSupportedDynamicPropertyDescriptor(final String
 .required(false)
 .dynamic(true)
 .addValidator(Validator.VALID)
+.expressionLanguageSupported(true)
--- End diff --

I meant to remove the expressionLanguageSupported(true). I don't think I 
had an especially good reason to have added the EL support to the dynamic 
property for the CS in the first place.


> Add lookup processor for enrichments/joins to reference data
> 
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining 
> against reference data sets or performing attribute lookups against external 
> data sources.
> Since enrichments and joins are basic streaming use cases, and since 
> attributes and EL are often used to parameterize processor properties, there 
> is a need for an easy way to do enrichments, joins and lookups without having 
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started 
> work on a LookupAttribute [2] processor that delegates the work to controller 
> services.
> 1. 
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E
> 2. https://github.com/jfrazee/nifi-lookup-service



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024094#comment-16024094
 ] 

ASF GitHub Bot commented on NIFI-3404:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1830#discussion_r118408138
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleKeyValueLookupService.java
 ---
@@ -47,6 +47,7 @@ protected PropertyDescriptor 
getSupportedDynamicPropertyDescriptor(final String
 .required(false)
 .dynamic(true)
 .addValidator(Validator.VALID)
+.expressionLanguageSupported(true)
--- End diff --

`cacheConfiguredValues` is still not evaluating expression. I can add EL 
evaluation when I merge.


> Add lookup processor for enrichments/joins to reference data
> 
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining 
> against reference data sets or performing attribute lookups against external 
> data sources.
> Since enrichments and joins are basic streaming use cases, and since 
> attributes and EL are often used to parameterize processor properties, there 
> is a need for an easy way to do enrichments, joins and lookups without having 
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started 
> work on a LookupAttribute [2] processor that delegates the work to controller 
> services.
> 1. 
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E
> 2. https://github.com/jfrazee/nifi-lookup-service



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024059#comment-16024059
 ] 

ASF subversion and git services commented on NIFI-3971:
---

Commit eaefec6d81f9dbf0d239776ef465b14109e28d18 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=eaefec6 ]

NIFI-3971: This closes #1854. Fixed bug in calculating content size that was 
transferred when cloning a relationship

Signed-off-by: joewitt 


> UpdateAttribute metric for output 'size' is larger than it should be when 
> cloning
> -
>
> Key: NIFI-3971
> URL: https://issues.apache.org/jira/browse/NIFI-3971
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.3.0
>
>
> When I configure UpdateAttribute so that the 'success' relationship goes to 
> multiple destinations, the output count is double what it should be. For 
> instance, if i send in 1 GB of data, the output size should be 2 GB (since 
> it's cloned to a second connection) but it shows 4 GB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt resolved NIFI-3971.
---
   Resolution: Fixed
Fix Version/s: 1.3.0

+1 merged to master

> UpdateAttribute metric for output 'size' is larger than it should be when 
> cloning
> -
>
> Key: NIFI-3971
> URL: https://issues.apache.org/jira/browse/NIFI-3971
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.3.0
>
>
> When I configure UpdateAttribute so that the 'success' relationship goes to 
> multiple destinations, the output count is double what it should be. For 
> instance, if i send in 1 GB of data, the output size should be 2 GB (since 
> it's cloned to a second connection) but it shows 4 GB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024060#comment-16024060
 ] 

ASF GitHub Bot commented on NIFI-3971:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1854


> UpdateAttribute metric for output 'size' is larger than it should be when 
> cloning
> -
>
> Key: NIFI-3971
> URL: https://issues.apache.org/jira/browse/NIFI-3971
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.3.0
>
>
> When I configure UpdateAttribute so that the 'success' relationship goes to 
> multiple destinations, the output count is double what it should be. For 
> instance, if i send in 1 GB of data, the output size should be 2 GB (since 
> it's cloned to a second connection) but it shows 4 GB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1854: NIFI-3971: Fixed bug in calculating content size th...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1854


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024058#comment-16024058
 ] 

ASF GitHub Bot commented on NIFI-3971:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/1854
  
+1 will merge to master.  thanks


> UpdateAttribute metric for output 'size' is larger than it should be when 
> cloning
> -
>
> Key: NIFI-3971
> URL: https://issues.apache.org/jira/browse/NIFI-3971
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> When I configure UpdateAttribute so that the 'success' relationship goes to 
> multiple destinations, the output count is double what it should be. For 
> instance, if i send in 1 GB of data, the output size should be 2 GB (since 
> it's cloned to a second connection) but it shows 4 GB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1854: NIFI-3971: Fixed bug in calculating content size that was ...

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/1854
  
+1 will merge to master.  thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-3972:
--
Priority: Critical  (was: Major)

> Controller Service failing to enabled because the service it depends on is 
> not fully enabled on nifi restart
> 
>
> Key: NIFI-3972
> URL: https://issues.apache.org/jira/browse/NIFI-3972
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.3.0
>
>
> I have a JsonRecordSetWriter controller service. It depends on a Schema 
> Registry controller service. Both are enabled. Upon restart, the 
> JsonRecordSetWriter service fails to enable, indicating that the Schema 
> Registry service is not enabled. When I look at the Controller Service 
> configuration, though, the Schema Registry service is enabled. So it appears 
> that the JsonRecordSetWriter was attempting to be enabled before the Schema 
> Registry Service finishes enabling.
> {code}
> 2017-05-24 19:46:56,314 ERROR [main] 
> o.a.n.c.s.StandardControllerServiceProvider Failed to enable 
> JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536]
> java.lang.IllegalStateException: Cannot invoke method public abstract 
> java.util.Set 
> org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields()
>  on Controller Service with identifier 0660348b-255e-1f3a--fb399783 
> because the Controller Service is disabled
> at 
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
> at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source)
> at 
> org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42)
> at 
> org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128)
> at 
> org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160)
> at 
> org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350)
> at 
> org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554)
> at 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722)
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
> at org.apache.nifi.NiFi.(NiFi.java:160)
> at org.apache.nifi.NiFi.main(NiFi.java:267)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024054#comment-16024054
 ] 

Joseph Witt commented on NIFI-3972:
---

setting fix version to 1.3.0.  the pr is close and this is a really important 
lifecycle condition to resolve

> Controller Service failing to enabled because the service it depends on is 
> not fully enabled on nifi restart
> 
>
> Key: NIFI-3972
> URL: https://issues.apache.org/jira/browse/NIFI-3972
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.3.0
>
>
> I have a JsonRecordSetWriter controller service. It depends on a Schema 
> Registry controller service. Both are enabled. Upon restart, the 
> JsonRecordSetWriter service fails to enable, indicating that the Schema 
> Registry service is not enabled. When I look at the Controller Service 
> configuration, though, the Schema Registry service is enabled. So it appears 
> that the JsonRecordSetWriter was attempting to be enabled before the Schema 
> Registry Service finishes enabling.
> {code}
> 2017-05-24 19:46:56,314 ERROR [main] 
> o.a.n.c.s.StandardControllerServiceProvider Failed to enable 
> JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536]
> java.lang.IllegalStateException: Cannot invoke method public abstract 
> java.util.Set 
> org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields()
>  on Controller Service with identifier 0660348b-255e-1f3a--fb399783 
> because the Controller Service is disabled
> at 
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
> at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source)
> at 
> org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42)
> at 
> org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128)
> at 
> org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160)
> at 
> org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350)
> at 
> org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554)
> at 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722)
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
> at org.apache.nifi.NiFi.(NiFi.java:160)
> at org.apache.nifi.NiFi.main(NiFi.java:267)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024053#comment-16024053
 ] 

ASF GitHub Bot commented on NIFI-3972:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/1855
  
initially it looked like it was sorted.  But after another round of 
restarting all nodes in the cluster the problem occurred again.  The case I 
have is a json record writer which depends on an external schema registry which 
could take a moment to enable and the writer is used by two kafka processors.  
The processors are not starting up as they should when the nodes startup 
because the writer service is disabled which is presumably because the service 
it depends on is not available.


> Controller Service failing to enabled because the service it depends on is 
> not fully enabled on nifi restart
> 
>
> Key: NIFI-3972
> URL: https://issues.apache.org/jira/browse/NIFI-3972
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.3.0
>
>
> I have a JsonRecordSetWriter controller service. It depends on a Schema 
> Registry controller service. Both are enabled. Upon restart, the 
> JsonRecordSetWriter service fails to enable, indicating that the Schema 
> Registry service is not enabled. When I look at the Controller Service 
> configuration, though, the Schema Registry service is enabled. So it appears 
> that the JsonRecordSetWriter was attempting to be enabled before the Schema 
> Registry Service finishes enabling.
> {code}
> 2017-05-24 19:46:56,314 ERROR [main] 
> o.a.n.c.s.StandardControllerServiceProvider Failed to enable 
> JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536]
> java.lang.IllegalStateException: Cannot invoke method public abstract 
> java.util.Set 
> org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields()
>  on Controller Service with identifier 0660348b-255e-1f3a--fb399783 
> because the Controller Service is disabled
> at 
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
> at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source)
> at 
> org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42)
> at 
> org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128)
> at 
> org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160)
> at 
> org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350)
> at 
> org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554)
> at 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84)
> at 
> 

[jira] [Updated] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-3972:
--
Fix Version/s: 1.3.0

> Controller Service failing to enabled because the service it depends on is 
> not fully enabled on nifi restart
> 
>
> Key: NIFI-3972
> URL: https://issues.apache.org/jira/browse/NIFI-3972
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.3.0
>
>
> I have a JsonRecordSetWriter controller service. It depends on a Schema 
> Registry controller service. Both are enabled. Upon restart, the 
> JsonRecordSetWriter service fails to enable, indicating that the Schema 
> Registry service is not enabled. When I look at the Controller Service 
> configuration, though, the Schema Registry service is enabled. So it appears 
> that the JsonRecordSetWriter was attempting to be enabled before the Schema 
> Registry Service finishes enabling.
> {code}
> 2017-05-24 19:46:56,314 ERROR [main] 
> o.a.n.c.s.StandardControllerServiceProvider Failed to enable 
> JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536]
> java.lang.IllegalStateException: Cannot invoke method public abstract 
> java.util.Set 
> org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields()
>  on Controller Service with identifier 0660348b-255e-1f3a--fb399783 
> because the Controller Service is disabled
> at 
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
> at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source)
> at 
> org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42)
> at 
> org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128)
> at 
> org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160)
> at 
> org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350)
> at 
> org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554)
> at 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722)
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
> at org.apache.nifi.NiFi.(NiFi.java:160)
> at org.apache.nifi.NiFi.main(NiFi.java:267)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1855: NIFI-3972: Ensure that we wait until service state becomes...

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/1855
  
initially it looked like it was sorted.  But after another round of 
restarting all nodes in the cluster the problem occurred again.  The case I 
have is a json record writer which depends on an external schema registry which 
could take a moment to enable and the writer is used by two kafka processors.  
The processors are not starting up as they should when the nodes startup 
because the writer service is disabled which is presumably because the service 
it depends on is not available.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024043#comment-16024043
 ] 

ASF GitHub Bot commented on NIFI-3404:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1830#discussion_r118404133
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleCsvFileLookupService.java
 ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.lookup;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.file.monitor.LastModifiedMonitor;
+import org.apache.nifi.util.file.monitor.SynchronousFileWatcher;
+
+@Tags({"lookup", "cache", "enrich", "join", "csv", "reloadable", "key", 
"value"})
+@CapabilityDescription("A reloadable properties file-based lookup service")
--- End diff --

"CSV file-based" instead of "properties".


> Add lookup processor for enrichments/joins to reference data
> 
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining 
> against reference data sets or performing attribute lookups against external 
> data sources.
> Since enrichments and joins are basic streaming use cases, and since 
> attributes and EL are often used to parameterize processor properties, there 
> is a need for an easy way to do enrichments, joins and lookups without having 
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started 
> work on a LookupAttribute [2] processor that delegates the work to controller 
> services.
> 1. 
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E
> 2. https://github.com/jfrazee/nifi-lookup-service



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1830#discussion_r118404133
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/SimpleCsvFileLookupService.java
 ---
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.lookup;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.file.monitor.LastModifiedMonitor;
+import org.apache.nifi.util.file.monitor.SynchronousFileWatcher;
+
+@Tags({"lookup", "cache", "enrich", "join", "csv", "reloadable", "key", 
"value"})
+@CapabilityDescription("A reloadable properties file-based lookup service")
--- End diff --

"CSV file-based" instead of "properties".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024025#comment-16024025
 ] 

ASF GitHub Bot commented on NIFI-3404:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1830
  
@jfrazee I understand, thank you! I am reviewing the updated commit now.


> Add lookup processor for enrichments/joins to reference data
> 
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining 
> against reference data sets or performing attribute lookups against external 
> data sources.
> Since enrichments and joins are basic streaming use cases, and since 
> attributes and EL are often used to parameterize processor properties, there 
> is a need for an easy way to do enrichments, joins and lookups without having 
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started 
> work on a LookupAttribute [2] processor that delegates the work to controller 
> services.
> 1. 
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E
> 2. https://github.com/jfrazee/nifi-lookup-service



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1830: NIFI-3404 Add lookup processor for enrichments/joins to re...

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1830
  
@jfrazee I understand, thank you! I am reviewing the updated commit now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024021#comment-16024021
 ] 

ASF GitHub Bot commented on NIFI-3404:
--

Github user jfrazee commented on the issue:

https://github.com/apache/nifi/pull/1830
  
@ijokarumawak The changes from 
[NIFI-3339](https://issues.apache.org/jira/browse/NIFI-3339)/#1450 have been 
removed from this PR along with the DatabaseLookupService. Will re-submit that 
stuff in another PR so we can move ahead with this and work out the details 
separately.


> Add lookup processor for enrichments/joins to reference data
> 
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining 
> against reference data sets or performing attribute lookups against external 
> data sources.
> Since enrichments and joins are basic streaming use cases, and since 
> attributes and EL are often used to parameterize processor properties, there 
> is a need for an easy way to do enrichments, joins and lookups without having 
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started 
> work on a LookupAttribute [2] processor that delegates the work to controller 
> services.
> 1. 
> https://lists.apache.org/thread.html/74321ff0e9e0b7339e43ad53b36119315dc5094991605edfb12b34d0@%3Cdev.nifi.apache.org%3E
> 2. https://github.com/jfrazee/nifi-lookup-service



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1830: NIFI-3404 Add lookup processor for enrichments/joins to re...

[jfrazee]

Github user jfrazee commented on the issue:

https://github.com/apache/nifi/pull/1830
  
@ijokarumawak The changes from 
[NIFI-3339](https://issues.apache.org/jira/browse/NIFI-3339)/#1450 have been 
removed from this PR along with the DatabaseLookupService. Will re-submit that 
stuff in another PR so we can move ahead with this and work out the details 
separately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024017#comment-16024017
 ] 

Joseph Witt commented on NIFI-3971:
---

under review

> UpdateAttribute metric for output 'size' is larger than it should be when 
> cloning
> -
>
> Key: NIFI-3971
> URL: https://issues.apache.org/jira/browse/NIFI-3971
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> When I configure UpdateAttribute so that the 'success' relationship goes to 
> multiple destinations, the output count is double what it should be. For 
> instance, if i send in 1 GB of data, the output size should be 2 GB (since 
> it's cloned to a second connection) but it shows 4 GB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3568) Default thread pool that is created for cluster request replication is not sufficient

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-3568:
--
Fix Version/s: 1.3.0

> Default thread pool that is created for cluster request replication is not 
> sufficient
> -
>
> Key: NIFI-3568
> URL: https://issues.apache.org/jira/browse/NIFI-3568
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.3.0
>
>
> I have a cluster of 3 nodes. When the nodes are under heavy load, I notice 
> that API requests sometimes complete in 10's to 100's of milliseconds but 
> sometimes take several seconds (8+ seconds, at times).
> After doing some investigation, it appears to be due to the fact that the 
> default thread pool size of 10 is not sufficient anymore. In the 0.x 
> baseline, it was okay because each time that a user clicks "Refresh" on the 
> UI it was a single request. With the 1.x baseline, this results in 4 separate 
> requests fired off simultaneously due to the multi-tenancy features added. As 
> a result, these 4 requests need to be replicated to 3 nodes each, which is 12 
> web requests that have to occur. So even a simple Refresh on the UI cannot be 
> fully done in parallel.
> Changing my pool size from 10 to 30 resulted in far more consistent response 
> times. Unfortunately, scaling the thread pool up to a large number of threads 
> can have its cons, too. So will create a "cached" thread pool and expose 
> properties for the "core pool size" and the "max pool size".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3568) Default thread pool that is created for cluster request replication is not sufficient

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-3568:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1 merged to master.  Cluster responsiveness was noticeably improved.

> Default thread pool that is created for cluster request replication is not 
> sufficient
> -
>
> Key: NIFI-3568
> URL: https://issues.apache.org/jira/browse/NIFI-3568
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> I have a cluster of 3 nodes. When the nodes are under heavy load, I notice 
> that API requests sometimes complete in 10's to 100's of milliseconds but 
> sometimes take several seconds (8+ seconds, at times).
> After doing some investigation, it appears to be due to the fact that the 
> default thread pool size of 10 is not sufficient anymore. In the 0.x 
> baseline, it was okay because each time that a user clicks "Refresh" on the 
> UI it was a single request. With the 1.x baseline, this results in 4 separate 
> requests fired off simultaneously due to the multi-tenancy features added. As 
> a result, these 4 requests need to be replicated to 3 nodes each, which is 12 
> web requests that have to occur. So even a simple Refresh on the UI cannot be 
> fully done in parallel.
> Changing my pool size from 10 to 30 resulted in far more consistent response 
> times. Unfortunately, scaling the thread pool up to a large number of threads 
> can have its cons, too. So will create a "cached" thread pool and expose 
> properties for the "core pool size" and the "max pool size".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1577: NIFI-3568: Use a cached thread pool in order to all...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1577


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3568) Default thread pool that is created for cluster request replication is not sufficient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023994#comment-16023994
 ] 

ASF GitHub Bot commented on NIFI-3568:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1577


> Default thread pool that is created for cluster request replication is not 
> sufficient
> -
>
> Key: NIFI-3568
> URL: https://issues.apache.org/jira/browse/NIFI-3568
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> I have a cluster of 3 nodes. When the nodes are under heavy load, I notice 
> that API requests sometimes complete in 10's to 100's of milliseconds but 
> sometimes take several seconds (8+ seconds, at times).
> After doing some investigation, it appears to be due to the fact that the 
> default thread pool size of 10 is not sufficient anymore. In the 0.x 
> baseline, it was okay because each time that a user clicks "Refresh" on the 
> UI it was a single request. With the 1.x baseline, this results in 4 separate 
> requests fired off simultaneously due to the multi-tenancy features added. As 
> a result, these 4 requests need to be replicated to 3 nodes each, which is 12 
> web requests that have to occur. So even a simple Refresh on the UI cannot be 
> fully done in parallel.
> Changing my pool size from 10 to 30 resulted in far more consistent response 
> times. Unfortunately, scaling the thread pool up to a large number of threads 
> can have its cons, too. So will create a "cached" thread pool and expose 
> properties for the "core pool size" and the "max pool size".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023990#comment-16023990
 ] 

Joseph Witt commented on NIFI-3972:
---

have this in review.  Looking much better.

> Controller Service failing to enabled because the service it depends on is 
> not fully enabled on nifi restart
> 
>
> Key: NIFI-3972
> URL: https://issues.apache.org/jira/browse/NIFI-3972
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> I have a JsonRecordSetWriter controller service. It depends on a Schema 
> Registry controller service. Both are enabled. Upon restart, the 
> JsonRecordSetWriter service fails to enable, indicating that the Schema 
> Registry service is not enabled. When I look at the Controller Service 
> configuration, though, the Schema Registry service is enabled. So it appears 
> that the JsonRecordSetWriter was attempting to be enabled before the Schema 
> Registry Service finishes enabling.
> {code}
> 2017-05-24 19:46:56,314 ERROR [main] 
> o.a.n.c.s.StandardControllerServiceProvider Failed to enable 
> JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536]
> java.lang.IllegalStateException: Cannot invoke method public abstract 
> java.util.Set 
> org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields()
>  on Controller Service with identifier 0660348b-255e-1f3a--fb399783 
> because the Controller Service is disabled
> at 
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
> at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source)
> at 
> org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42)
> at 
> org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128)
> at 
> org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160)
> at 
> org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350)
> at 
> org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554)
> at 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722)
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
> at org.apache.nifi.NiFi.(NiFi.java:160)
> at org.apache.nifi.NiFi.main(NiFi.java:267)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (NIFI-3973) Create a new Kudu Processor to ingest data

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne reassigned NIFI-3973:


Assignee: Cam Quoc Mach

> Create a new Kudu Processor to ingest data
> --
>
> Key: NIFI-3973
> URL: https://issues.apache.org/jira/browse/NIFI-3973
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Cam Mach
>Assignee: Cam Quoc Mach
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3973) Create a new Kudu Processor to ingest data

[Cam Mach (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023958#comment-16023958
 ] 

Cam Mach commented on NIFI-3973:


Hi, I would like to contribute my Kudu processor to the project. Please review 
and let me know what need to be adjusted.

> Create a new Kudu Processor to ingest data
> --
>
> Key: NIFI-3973
> URL: https://issues.apache.org/jira/browse/NIFI-3973
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Cam Mach
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3973) Create a new Kudu Processor to ingest data

[Cam Mach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cam Mach updated NIFI-3973:
---
Summary: Create a new Kudu Processor to ingest data  (was: Create a new 
Kudu Process to ingest data)

> Create a new Kudu Processor to ingest data
> --
>
> Key: NIFI-3973
> URL: https://issues.apache.org/jira/browse/NIFI-3973
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Cam Mach
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3973) Create a new Kudu Process to ingest data

[Cam Mach (JIRA)]

Cam Mach created NIFI-3973:
--

 Summary: Create a new Kudu Process to ingest data
 Key: NIFI-3973
 URL: https://issues.apache.org/jira/browse/NIFI-3973
 Project: Apache NiFi
  Issue Type: New Feature
  Components: Extensions
Reporter: Cam Mach






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi-minifi-cpp issue #104: MINIFI-296 - Configurable logging, spdlog 0.13.0

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/104
  
@brosander yeah I think an atomic would solve the problem. 

yeah, I think the pattern would be cool. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #104: MINIFI-296 - Configurable logging, spdlog 0.13.0

[brosander]

Github user brosander commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/104
  
@phrocker 

For your first point about reinitializing logging while a flow is running, 
is it worth it to switch the delegate reference on the Logger to an atomic?

Second point I think should be fairly easily addressable with a format 
pattern in the logger config.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3925) Add Template list with long template name

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-3925:
--
   Resolution: Fixed
Fix Version/s: 1.3.0
   Status: Resolved  (was: Patch Available)

> Add Template list with long template name
> -
>
> Key: NIFI-3925
> URL: https://issues.apache.org/jira/browse/NIFI-3925
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.2.0
>Reporter: Mark Bean
>Assignee: Scott Aslan
>Priority: Minor
> Fix For: 1.3.0
>
>
> If a template has a very long name (roughly 40 characters or more), the 
> dropdown list of templates in "Add Template" does not display properly. The 
> "?" rollover which displays the template description is shown on the next 
> line. In effect, the long-named template line has no "?" rollover, and the 
> next template line has two adjacent "?" rollovers (one corresponding to the 
> previous, long-named template and one corresponding to itself.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3925) Add Template list with long template name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023617#comment-16023617
 ] 

ASF GitHub Bot commented on NIFI-3925:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1848
  
Thanks @scottyaslan. This has been merged to master.


> Add Template list with long template name
> -
>
> Key: NIFI-3925
> URL: https://issues.apache.org/jira/browse/NIFI-3925
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.2.0
>Reporter: Mark Bean
>Assignee: Scott Aslan
>Priority: Minor
>
> If a template has a very long name (roughly 40 characters or more), the 
> dropdown list of templates in "Add Template" does not display properly. The 
> "?" rollover which displays the template description is shown on the next 
> line. In effect, the long-named template line has no "?" rollover, and the 
> next template line has two adjacent "?" rollovers (one corresponding to the 
> previous, long-named template and one corresponding to itself.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1848: [NIFI-3925] bound width of combo options drop down text

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1848
  
Thanks @scottyaslan. This has been merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3925) Add Template list with long template name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023612#comment-16023612
 ] 

ASF GitHub Bot commented on NIFI-3925:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1848


> Add Template list with long template name
> -
>
> Key: NIFI-3925
> URL: https://issues.apache.org/jira/browse/NIFI-3925
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.2.0
>Reporter: Mark Bean
>Assignee: Scott Aslan
>Priority: Minor
>
> If a template has a very long name (roughly 40 characters or more), the 
> dropdown list of templates in "Add Template" does not display properly. The 
> "?" rollover which displays the template description is shown on the next 
> line. In effect, the long-named template line has no "?" rollover, and the 
> next template line has two adjacent "?" rollovers (one corresponding to the 
> previous, long-named template and one corresponding to itself.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3925) Add Template list with long template name

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023609#comment-16023609
 ] 

ASF subversion and git services commented on NIFI-3925:
---

Commit 86728bac7ef091dba79cd3ffd3219c80f4fca8ba in nifi's branch 
refs/heads/master from [~scottyaslan]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=86728ba ]

[NIFI-3925] for the jquery.combo plugin: update the calculation of the width of 
the combo box options overlay and also updated the plugins styles to now 
leverage the css calc() to determine the exact width of .combotext and 
.combo-option-text elements. This closes #1848


> Add Template list with long template name
> -
>
> Key: NIFI-3925
> URL: https://issues.apache.org/jira/browse/NIFI-3925
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.2.0
>Reporter: Mark Bean
>Assignee: Scott Aslan
>Priority: Minor
>
> If a template has a very long name (roughly 40 characters or more), the 
> dropdown list of templates in "Add Template" does not display properly. The 
> "?" rollover which displays the template description is shown on the next 
> line. In effect, the long-named template line has no "?" rollover, and the 
> next template line has two adjacent "?" rollovers (one corresponding to the 
> previous, long-named template and one corresponding to itself.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1848: [NIFI-3925] bound width of combo options drop down ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1848


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #104: MINIFI-296 - Configurable logging, spdlog 0.13.0

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/104
  
This is good stuff. I'm taking a look

General comments before I dive in. Did we lose the ability to change the 
logger format real time? If there was an issue in production and we had to 
change the logger to a null appender, restarting may not always be possible, 
whereas sending a shared object through a p2p c2 protocol would allow us to 
dynamically load and change loggers without impacting the implementation, for 
example ( kind of a contrived example)

Additionally, I Noticed that my logs rolled much more frequently because we 
have an additional 52 bytes per line in some cases. Is there a way to limit 
that? It slowed my runs down because there was much more memory allocated by 
spdlog log statements when threading was high and I tested with trace. Even to 
the class name would be useful. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-3972:
-
Status: Patch Available  (was: Open)

> Controller Service failing to enabled because the service it depends on is 
> not fully enabled on nifi restart
> 
>
> Key: NIFI-3972
> URL: https://issues.apache.org/jira/browse/NIFI-3972
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> I have a JsonRecordSetWriter controller service. It depends on a Schema 
> Registry controller service. Both are enabled. Upon restart, the 
> JsonRecordSetWriter service fails to enable, indicating that the Schema 
> Registry service is not enabled. When I look at the Controller Service 
> configuration, though, the Schema Registry service is enabled. So it appears 
> that the JsonRecordSetWriter was attempting to be enabled before the Schema 
> Registry Service finishes enabling.
> {code}
> 2017-05-24 19:46:56,314 ERROR [main] 
> o.a.n.c.s.StandardControllerServiceProvider Failed to enable 
> JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536]
> java.lang.IllegalStateException: Cannot invoke method public abstract 
> java.util.Set 
> org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields()
>  on Controller Service with identifier 0660348b-255e-1f3a--fb399783 
> because the Controller Service is disabled
> at 
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
> at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source)
> at 
> org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42)
> at 
> org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128)
> at 
> org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160)
> at 
> org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326)
> at 
> org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384)
> at 
> org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350)
> at 
> org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159)
> at 
> org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042)
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312)
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554)
> at 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84)
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722)
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
> at org.apache.nifi.NiFi.(NiFi.java:160)
> at org.apache.nifi.NiFi.main(NiFi.java:267)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023606#comment-16023606
 ] 

ASF GitHub Bot commented on NIFI-3972:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1855

NIFI-3972: Ensure that we wait until service state becomes enabled be…

…fore triggering completable future that says that it's enabled

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3972

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1855.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1855


commit 27ca0d809e3105993396d80b7efb5268f68a263d
Author: Mark Payne 
Date:   2017-05-24T20:15:18Z

NIFI-3972: Ensure that we wait until service state becomes enabled before 
triggering completable future that says that it's enabled




> Controller Service failing to enabled because the service it depends on is 
> not fully enabled on nifi restart
> 
>
> Key: NIFI-3972
> URL: https://issues.apache.org/jira/browse/NIFI-3972
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> I have a JsonRecordSetWriter controller service. It depends on a Schema 
> Registry controller service. Both are enabled. Upon restart, the 
> JsonRecordSetWriter service fails to enable, indicating that the Schema 
> Registry service is not enabled. When I look at the Controller Service 
> configuration, though, the Schema Registry service is enabled. So it appears 
> that the JsonRecordSetWriter was attempting to be enabled before the Schema 
> Registry Service finishes enabling.
> {code}
> 2017-05-24 19:46:56,314 ERROR [main] 
> o.a.n.c.s.StandardControllerServiceProvider Failed to enable 
> JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536]
> java.lang.IllegalStateException: Cannot invoke method public abstract 
> java.util.Set 
> org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields()
>  on Controller Service with identifier 0660348b-255e-1f3a--fb399783 
> because the Controller Service is disabled
> at 
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
> at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source)
> at 
> org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42)
> at 
> org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147)
> at 
> org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149)
> at 
> 

[GitHub] nifi pull request #1855: NIFI-3972: Ensure that we wait until service state ...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1855

NIFI-3972: Ensure that we wait until service state becomes enabled be…

…fore triggering completable future that says that it's enabled

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3972

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1855.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1855


commit 27ca0d809e3105993396d80b7efb5268f68a263d
Author: Mark Payne 
Date:   2017-05-24T20:15:18Z

NIFI-3972: Ensure that we wait until service state becomes enabled before 
triggering completable future that says that it's enabled




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-3972) Controller Service failing to enabled because the service it depends on is not fully enabled on nifi restart

[Mark Payne (JIRA)]

Mark Payne created NIFI-3972:


 Summary: Controller Service failing to enabled because the service 
it depends on is not fully enabled on nifi restart
 Key: NIFI-3972
 URL: https://issues.apache.org/jira/browse/NIFI-3972
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne


I have a JsonRecordSetWriter controller service. It depends on a Schema 
Registry controller service. Both are enabled. Upon restart, the 
JsonRecordSetWriter service fails to enable, indicating that the Schema 
Registry service is not enabled. When I look at the Controller Service 
configuration, though, the Schema Registry service is enabled. So it appears 
that the JsonRecordSetWriter was attempting to be enabled before the Schema 
Registry Service finishes enabling.

{code}
2017-05-24 19:46:56,314 ERROR [main] 
o.a.n.c.s.StandardControllerServiceProvider Failed to enable 
JsonRecordSetWriter[id=0d273e88-9bd7-1dd9--c1e1a536]
java.lang.IllegalStateException: Cannot invoke method public abstract 
java.util.Set 
org.apache.nifi.schemaregistry.services.SchemaRegistry.getSuppliedSchemaFields()
 on Controller Service with identifier 0660348b-255e-1f3a--fb399783 
because the Controller Service is disabled
at 
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
at com.sun.proxy.$Proxy85.getSuppliedSchemaFields(Unknown Source)
at 
org.apache.nifi.schema.access.SchemaNamePropertyStrategy.(SchemaNamePropertyStrategy.java:42)
at 
org.apache.nifi.schema.access.SchemaAccessUtils.getSchemaAccessStrategy(SchemaAccessUtils.java:147)
at 
org.apache.nifi.serialization.SchemaRegistryService.getSchemaAccessStrategy(SchemaRegistryService.java:149)
at 
org.apache.nifi.serialization.SchemaRegistryService.getSuppliedSchemaFields(SchemaRegistryService.java:128)
at 
org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.customValidate(SchemaRegistryRecordSetWriter.java:160)
at 
org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:126)
at 
org.apache.nifi.controller.AbstractConfiguredComponent.validate(AbstractConfiguredComponent.java:326)
at 
org.apache.nifi.controller.AbstractConfiguredComponent.isValid(AbstractConfiguredComponent.java:441)
at 
org.apache.nifi.controller.service.StandardControllerServiceNode.verifyCanEnable(StandardControllerServiceNode.java:301)
at 
org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerService(StandardControllerServiceProvider.java:327)
at 
org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServiceDependenciesFirst(StandardControllerServiceProvider.java:384)
at 
org.apache.nifi.controller.service.StandardControllerServiceProvider.enableControllerServices(StandardControllerServiceProvider.java:350)
at 
org.apache.nifi.controller.FlowController.enableControllerServices(FlowController.java:3271)
at 
org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:159)
at 
org.apache.nifi.controller.service.ControllerServiceLoader.enableControllerServices(ControllerServiceLoader.java:152)
at 
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1042)
at 
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:312)
at 
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1554)
at 
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84)
at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722)
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:452)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
at org.apache.nifi.NiFi.(NiFi.java:160)
at org.apache.nifi.NiFi.main(NiFi.java:267)
{code}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende updated NIFI-3963:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Remote Process Group does not honor yield duration if transmission is 
> disabled then reenabled
> -
>
> Key: NIFI-3963
> URL: https://issues.apache.org/jira/browse/NIFI-3963
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Joseph Percivall
>Assignee: Matt Gilman
> Fix For: 1.3.0
>
> Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, 
> unable-to-reproduce-nifi-3963.png
>
>
> To reproduce, create two NiFi instances properly set up for unsecure or 
> secure  S2S. One on side add an input/output port and the other an RPG. 
> Enable transmission on the RPG. Stop the instance with the port. On the 
> running instance see NiFi honoring the yield duration. Then stop and start 
> transmission, see NiFi repeatedly attempt to connect to the instance (I saw 
> multiple log messages per millisecond). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023568#comment-16023568
 ] 

ASF GitHub Bot commented on NIFI-3963:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1853


> Remote Process Group does not honor yield duration if transmission is 
> disabled then reenabled
> -
>
> Key: NIFI-3963
> URL: https://issues.apache.org/jira/browse/NIFI-3963
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Joseph Percivall
>Assignee: Matt Gilman
> Fix For: 1.3.0
>
> Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, 
> unable-to-reproduce-nifi-3963.png
>
>
> To reproduce, create two NiFi instances properly set up for unsecure or 
> secure  S2S. One on side add an input/output port and the other an RPG. 
> Enable transmission on the RPG. Stop the instance with the port. On the 
> running instance see NiFi honoring the yield duration. Then stop and start 
> transmission, see NiFi repeatedly attempt to connect to the instance (I saw 
> multiple log messages per millisecond). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1853: NIFI-3963: RPG Yield Issue

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1853


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023563#comment-16023563
 ] 

ASF GitHub Bot commented on NIFI-3963:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1853
  
+1 Looks good, repeated the steps outline by Joe P and can see that we are 
now yielding correctly, will merge to master


> Remote Process Group does not honor yield duration if transmission is 
> disabled then reenabled
> -
>
> Key: NIFI-3963
> URL: https://issues.apache.org/jira/browse/NIFI-3963
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Joseph Percivall
>Assignee: Matt Gilman
> Fix For: 1.3.0
>
> Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, 
> unable-to-reproduce-nifi-3963.png
>
>
> To reproduce, create two NiFi instances properly set up for unsecure or 
> secure  S2S. One on side add an input/output port and the other an RPG. 
> Enable transmission on the RPG. Stop the instance with the port. On the 
> running instance see NiFi honoring the yield duration. Then stop and start 
> transmission, see NiFi repeatedly attempt to connect to the instance (I saw 
> multiple log messages per millisecond). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1853: NIFI-3963: RPG Yield Issue

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1853
  
+1 Looks good, repeated the steps outline by Joe P and can see that we are 
now yielding correctly, will merge to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023511#comment-16023511
 ] 

ASF GitHub Bot commented on NIFI-3971:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1854

NIFI-3971: Fixed bug in calculating content size that was transferred…

… when cloning a relationship

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3971

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1854.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1854


commit 8d4bc8a2301535c6ca6041bda3c0ab35ee7531ea
Author: Mark Payne 
Date:   2017-05-24T19:31:33Z

NIFI-3971: Fixed bug in calculating content size that was transferred when 
cloning a relationship




> UpdateAttribute metric for output 'size' is larger than it should be when 
> cloning
> -
>
> Key: NIFI-3971
> URL: https://issues.apache.org/jira/browse/NIFI-3971
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> When I configure UpdateAttribute so that the 'success' relationship goes to 
> multiple destinations, the output count is double what it should be. For 
> instance, if i send in 1 GB of data, the output size should be 2 GB (since 
> it's cloned to a second connection) but it shows 4 GB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1854: NIFI-3971: Fixed bug in calculating content size th...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1854

NIFI-3971: Fixed bug in calculating content size that was transferred…

… when cloning a relationship

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3971

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1854.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1854


commit 8d4bc8a2301535c6ca6041bda3c0ab35ee7531ea
Author: Mark Payne 
Date:   2017-05-24T19:31:33Z

NIFI-3971: Fixed bug in calculating content size that was transferred when 
cloning a relationship




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-3970) Add CSVRecordLookupService

[Joey Frazee (JIRA)]

Joey Frazee created NIFI-3970:
-

 Summary: Add CSVRecordLookupService
 Key: NIFI-3970
 URL: https://issues.apache.org/jira/browse/NIFI-3970
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Joey Frazee
Assignee: Joey Frazee


PR [#1830|https://github.com/apache/nifi/pull/1830] provides a 
SimpleCsvFileLookupService. Since CSV data is tabular, a counterpart 
CSVRecordLookupService would be useful for using with multi-criteria lookups 
and enrichments in LookupRecord and LookupAttribute.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3971) UpdateAttribute metric for output 'size' is larger than it should be when cloning

[Mark Payne (JIRA)]

Mark Payne created NIFI-3971:


 Summary: UpdateAttribute metric for output 'size' is larger than 
it should be when cloning
 Key: NIFI-3971
 URL: https://issues.apache.org/jira/browse/NIFI-3971
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne


When I configure UpdateAttribute so that the 'success' relationship goes to 
multiple destinations, the output count is double what it should be. For 
instance, if i send in 1 GB of data, the output size should be 2 GB (since it's 
cloned to a second connection) but it shows 4 GB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi-minifi-cpp pull request #104: MINIFI-296 - Configurable logging, spdlog...

[brosander]

GitHub user brosander opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/104

MINIFI-296 - Configurable logging, spdlog 0.13.0

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [X] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] - N/A - If applicable, have you updated the LICENSE file?
- [X] - N/A - If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [X] - N/A - Have you ensured that format looks appropriate for the output 
in which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brosander/nifi-minifi-cpp MINIFI-296

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/104.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #104


commit fa0304cd48920a1d3b259b9a51a6aff0c79a88e7
Author: Bryan Rosander 
Date:   2017-05-11T19:42:54Z

MINIFI-296 - Configurable logging, spdlog 0.13.0




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023483#comment-16023483
 ] 

ASF GitHub Bot commented on NIFI-3963:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1853
  
Reviewing...


> Remote Process Group does not honor yield duration if transmission is 
> disabled then reenabled
> -
>
> Key: NIFI-3963
> URL: https://issues.apache.org/jira/browse/NIFI-3963
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Joseph Percivall
>Assignee: Matt Gilman
> Fix For: 1.3.0
>
> Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, 
> unable-to-reproduce-nifi-3963.png
>
>
> To reproduce, create two NiFi instances properly set up for unsecure or 
> secure  S2S. One on side add an input/output port and the other an RPG. 
> Enable transmission on the RPG. Stop the instance with the port. On the 
> running instance see NiFi honoring the yield duration. Then stop and start 
> transmission, see NiFi repeatedly attempt to connect to the instance (I saw 
> multiple log messages per millisecond). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1853: NIFI-3963: RPG Yield Issue

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1853
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3404) Add lookup processor for enrichments/joins to reference data

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023479#comment-16023479
 ] 

ASF GitHub Bot commented on NIFI-3404:
--

Github user jfrazee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1830#discussion_r118341314
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java
 ---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes", 
"Attribute Expression Language"})
--- End diff --

Holdover from the interfaces this was originally written against. There was 
a LookupService and a MutableLookupService. Will remove.


> Add lookup processor for enrichments/joins to reference data
> 
>
> Key: NIFI-3404
> URL: https://issues.apache.org/jira/browse/NIFI-3404
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joey Frazee
>Assignee: Joey Frazee
>
> NiFi doesn't currently have an easy, concise way of doing enrichment, joining 
> against reference data sets or performing attribute lookups against external 
> data sources.
> Since enrichments and joins are basic streaming use cases, and since 
> attributes and EL are often used to parameterize processor properties, there 
> is a need for an easy way to do enrichments, joins and lookups without having 
> to write code or create a lengthy data flow.
> There's been some discussion of this on the mailing list [1] and I've started 
> work on a LookupAttribute [2] processor that delegates the work to controller 
> services.
> 1. 
> 

[GitHub] nifi pull request #1830: NIFI-3404 Add lookup processor for enrichments/join...

[jfrazee]

Github user jfrazee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1830#discussion_r118341314
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupAttribute.java
 ---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.lookup.LookupFailureException;
+import org.apache.nifi.lookup.LookupService;
+import org.apache.nifi.lookup.StringLookupService;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"lookup", "cache", "enrich", "join", "mutable", "attributes", 
"Attribute Expression Language"})
--- End diff --

Holdover from the interfaces this was originally written against. There was 
a LookupService and a MutableLookupService. Will remove.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3568) Default thread pool that is created for cluster request replication is not sufficient

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023463#comment-16023463
 ] 

Mark Payne commented on NIFI-3568:
--

[~joewitt] - rebased. Thanks!

> Default thread pool that is created for cluster request replication is not 
> sufficient
> -
>
> Key: NIFI-3568
> URL: https://issues.apache.org/jira/browse/NIFI-3568
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> I have a cluster of 3 nodes. When the nodes are under heavy load, I notice 
> that API requests sometimes complete in 10's to 100's of milliseconds but 
> sometimes take several seconds (8+ seconds, at times).
> After doing some investigation, it appears to be due to the fact that the 
> default thread pool size of 10 is not sufficient anymore. In the 0.x 
> baseline, it was okay because each time that a user clicks "Refresh" on the 
> UI it was a single request. With the 1.x baseline, this results in 4 separate 
> requests fired off simultaneously due to the multi-tenancy features added. As 
> a result, these 4 requests need to be replicated to 3 nodes each, which is 12 
> web requests that have to occur. So even a simple Refresh on the UI cannot be 
> fully done in parallel.
> Changing my pool size from 10 to 30 resulted in far more consistent response 
> times. Unfortunately, scaling the thread pool up to a large number of threads 
> can have its cons, too. So will create a "cached" thread pool and expose 
> properties for the "core pool size" and the "max pool size".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3644) Add DetectDuplicateUsingHBase processor

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023456#comment-16023456
 ] 

Joseph Witt commented on NIFI-3644:
---

[~bjorn.ols...@gmail.com] and [~bbende] this is a really cool addition.  Nice 
work and thanks!

> Add DetectDuplicateUsingHBase processor
> ---
>
> Key: NIFI-3644
> URL: https://issues.apache.org/jira/browse/NIFI-3644
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Assignee: Bryan Bende
>Priority: Minor
> Fix For: 1.3.0
>
>
> The DetectDuplicate processor makes use of a distributed map cache for 
> maintaining a list of unique file identifiers (such as hashes).
> The distributed map cache functionality could be provided by an HBase table, 
> which then allows for reliably storing a huge volume of file identifiers and 
> auditing information. The downside of this approach is of course that HBase 
> is required.
> Storing the unique file identifiers in a reliable, query-able manner along 
> with some audit information is of benefit to several use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (NIFI-3644) Add DetectDuplicateUsingHBase processor

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende resolved NIFI-3644.
---
Resolution: Fixed
  Assignee: Bryan Bende

Merged to master.

> Add DetectDuplicateUsingHBase processor
> ---
>
> Key: NIFI-3644
> URL: https://issues.apache.org/jira/browse/NIFI-3644
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Assignee: Bryan Bende
>Priority: Minor
> Fix For: 1.3.0
>
>
> The DetectDuplicate processor makes use of a distributed map cache for 
> maintaining a list of unique file identifiers (such as hashes).
> The distributed map cache functionality could be provided by an HBase table, 
> which then allows for reliably storing a huge volume of file identifiers and 
> auditing information. The downside of this approach is of course that HBase 
> is required.
> Storing the unique file identifiers in a reliable, query-able manner along 
> with some audit information is of benefit to several use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3644) Add DetectDuplicateUsingHBase processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023451#comment-16023451
 ] 

ASF GitHub Bot commented on NIFI-3644:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1645


> Add DetectDuplicateUsingHBase processor
> ---
>
> Key: NIFI-3644
> URL: https://issues.apache.org/jira/browse/NIFI-3644
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Minor
> Fix For: 1.3.0
>
>
> The DetectDuplicate processor makes use of a distributed map cache for 
> maintaining a list of unique file identifiers (such as hashes).
> The distributed map cache functionality could be provided by an HBase table, 
> which then allows for reliably storing a huge volume of file identifiers and 
> auditing information. The downside of this approach is of course that HBase 
> is required.
> Storing the unique file identifiers in a reliable, query-able manner along 
> with some audit information is of benefit to several use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3644) Add DetectDuplicateUsingHBase processor

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende updated NIFI-3644:
--
Fix Version/s: 1.3.0

> Add DetectDuplicateUsingHBase processor
> ---
>
> Key: NIFI-3644
> URL: https://issues.apache.org/jira/browse/NIFI-3644
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Minor
> Fix For: 1.3.0
>
>
> The DetectDuplicate processor makes use of a distributed map cache for 
> maintaining a list of unique file identifiers (such as hashes).
> The distributed map cache functionality could be provided by an HBase table, 
> which then allows for reliably storing a huge volume of file identifiers and 
> auditing information. The downside of this approach is of course that HBase 
> is required.
> Storing the unique file identifiers in a reliable, query-able manner along 
> with some audit information is of benefit to several use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1645: NIFI-3644 - Added HBase_1_1_2_ClientMapCacheService

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1645


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3644) Add DetectDuplicateUsingHBase processor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023450#comment-16023450
 ] 

ASF subversion and git services commented on NIFI-3644:
---

Commit ae3db823037ef01f8dc123e494f1d9e6522f29fe in nifi's branch 
refs/heads/master from [~bbende]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ae3db82 ]

NIFI-3644 Fixing the result handler in HBase_1_1_2_ClientMapCacheService to use 
the offsets for the value bytes

This closes #1645.

Signed-off-by: Bryan Bende 


> Add DetectDuplicateUsingHBase processor
> ---
>
> Key: NIFI-3644
> URL: https://issues.apache.org/jira/browse/NIFI-3644
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Minor
>
> The DetectDuplicate processor makes use of a distributed map cache for 
> maintaining a list of unique file identifiers (such as hashes).
> The distributed map cache functionality could be provided by an HBase table, 
> which then allows for reliably storing a huge volume of file identifiers and 
> auditing information. The downside of this approach is of course that HBase 
> is required.
> Storing the unique file identifiers in a reliable, query-able manner along 
> with some audit information is of benefit to several use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3568) Default thread pool that is created for cluster request replication is not sufficient

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023445#comment-16023445
 ] 

Joseph Witt commented on NIFI-3568:
---

[~markap14] i've seen this have really good results on a high load cluster.  
Could you please rebase this so we can include it in the upcoming release?

> Default thread pool that is created for cluster request replication is not 
> sufficient
> -
>
> Key: NIFI-3568
> URL: https://issues.apache.org/jira/browse/NIFI-3568
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> I have a cluster of 3 nodes. When the nodes are under heavy load, I notice 
> that API requests sometimes complete in 10's to 100's of milliseconds but 
> sometimes take several seconds (8+ seconds, at times).
> After doing some investigation, it appears to be due to the fact that the 
> default thread pool size of 10 is not sufficient anymore. In the 0.x 
> baseline, it was okay because each time that a user clicks "Refresh" on the 
> UI it was a single request. With the 1.x baseline, this results in 4 separate 
> requests fired off simultaneously due to the multi-tenancy features added. As 
> a result, these 4 requests need to be replicated to 3 nodes each, which is 12 
> web requests that have to occur. So even a simple Refresh on the UI cannot be 
> fully done in parallel.
> Changing my pool size from 10 to 30 resulted in far more consistent response 
> times. Unfortunately, scaling the thread pool up to a large number of threads 
> can have its cons, too. So will create a "cached" thread pool and expose 
> properties for the "core pool size" and the "max pool size".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023410#comment-16023410
 ] 

ASF GitHub Bot commented on NIFI-3963:
--

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/1853

NIFI-3963: RPG Yield Issue

NIFI-3963:
- Ensuring the RemoteGroupPort yields when the details cannot be refreshed 
from any of the configured remote instances.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-3963

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1853.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1853


commit 12fd5f942df1531b28a25bd2eb369d58d7677934
Author: Matt Gilman 
Date:   2017-05-24T18:28:51Z

NIFI-3963:
- Ensuring the RemoteGroupPort yields when the details cannot be refreshed 
from any of the configured remote instances.




> Remote Process Group does not honor yield duration if transmission is 
> disabled then reenabled
> -
>
> Key: NIFI-3963
> URL: https://issues.apache.org/jira/browse/NIFI-3963
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Joseph Percivall
>Assignee: Matt Gilman
> Fix For: 1.3.0
>
> Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, 
> unable-to-reproduce-nifi-3963.png
>
>
> To reproduce, create two NiFi instances properly set up for unsecure or 
> secure  S2S. One on side add an input/output port and the other an RPG. 
> Enable transmission on the RPG. Stop the instance with the port. On the 
> running instance see NiFi honoring the yield duration. Then stop and start 
> transmission, see NiFi repeatedly attempt to connect to the instance (I saw 
> multiple log messages per millisecond). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3963) Remote Process Group does not honor yield duration if transmission is disabled then reenabled

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-3963:
--
Fix Version/s: 1.3.0
   Status: Patch Available  (was: In Progress)

> Remote Process Group does not honor yield duration if transmission is 
> disabled then reenabled
> -
>
> Key: NIFI-3963
> URL: https://issues.apache.org/jira/browse/NIFI-3963
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Joseph Percivall
>Assignee: Matt Gilman
> Fix For: 1.3.0
>
> Attachments: Screen Shot 2017-05-23 at 5.07.45 PM.png, 
> unable-to-reproduce-nifi-3963.png
>
>
> To reproduce, create two NiFi instances properly set up for unsecure or 
> secure  S2S. One on side add an input/output port and the other an RPG. 
> Enable transmission on the RPG. Stop the instance with the port. On the 
> running instance see NiFi honoring the yield duration. Then stop and start 
> transmission, see NiFi repeatedly attempt to connect to the instance (I saw 
> multiple log messages per millisecond). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-1452) Yield Duration can short circuit long Timer Driven Run Schedule

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-1452:
-
   Resolution: Fixed
Fix Version/s: 1.3.0
   Status: Resolved  (was: Patch Available)

> Yield Duration can short circuit long Timer Driven Run Schedule
> ---
>
> Key: NIFI-1452
> URL: https://issues.apache.org/jira/browse/NIFI-1452
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.4.1
>Reporter: Michael Moser
>Assignee: Michael Moser
>Priority: Trivial
> Fix For: 1.3.0
>
>
> This may be a rare use case, but I configured a GetFile processor to be Timer 
> Driven with a Run Schedule of 30 secs.  Its Yield Duration was default 1 sec. 
>  I expected GetFile onTrigger() to be called every 30 secs, but it was being 
> called every 1 sec most of the time.
> GetFile will call context.yield() when it polls a directory and gets an empty 
> list in return.  It appears that a yield will ignore the Run Schedule.  Many 
> standard processors call context.yield() when they have no work to do.
> I changed my scheduling strategy to CRON Driven with its run schedule every 
> 30 seconds, and the onTrigger() was called every 30 seconds, even after a 
> yield.  So CRON Driven scheduling is working as expected after a yield.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-1452) Yield Duration can short circuit long Timer Driven Run Schedule

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023395#comment-16023395
 ] 

ASF GitHub Bot commented on NIFI-1452:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1832
  
Was able to test and verify functionality. +1 merged to master. Thanks 
@mosermw!


> Yield Duration can short circuit long Timer Driven Run Schedule
> ---
>
> Key: NIFI-1452
> URL: https://issues.apache.org/jira/browse/NIFI-1452
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.4.1
>Reporter: Michael Moser
>Assignee: Michael Moser
>Priority: Trivial
>
> This may be a rare use case, but I configured a GetFile processor to be Timer 
> Driven with a Run Schedule of 30 secs.  Its Yield Duration was default 1 sec. 
>  I expected GetFile onTrigger() to be called every 30 secs, but it was being 
> called every 1 sec most of the time.
> GetFile will call context.yield() when it polls a directory and gets an empty 
> list in return.  It appears that a yield will ignore the Run Schedule.  Many 
> standard processors call context.yield() when they have no work to do.
> I changed my scheduling strategy to CRON Driven with its run schedule every 
> 30 seconds, and the onTrigger() was called every 30 seconds, even after a 
> yield.  So CRON Driven scheduling is working as expected after a yield.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1832: NIFI-1452 on timer-driven yield, use the greater of...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1832


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1832: NIFI-1452 on timer-driven yield, use the greater of yield ...

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1832
  
Was able to test and verify functionality. +1 merged to master. Thanks 
@mosermw!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1452) Yield Duration can short circuit long Timer Driven Run Schedule

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023393#comment-16023393
 ] 

ASF GitHub Bot commented on NIFI-1452:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1832


> Yield Duration can short circuit long Timer Driven Run Schedule
> ---
>
> Key: NIFI-1452
> URL: https://issues.apache.org/jira/browse/NIFI-1452
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.4.1
>Reporter: Michael Moser
>Assignee: Michael Moser
>Priority: Trivial
>
> This may be a rare use case, but I configured a GetFile processor to be Timer 
> Driven with a Run Schedule of 30 secs.  Its Yield Duration was default 1 sec. 
>  I expected GetFile onTrigger() to be called every 30 secs, but it was being 
> called every 1 sec most of the time.
> GetFile will call context.yield() when it polls a directory and gets an empty 
> list in return.  It appears that a yield will ignore the Run Schedule.  Many 
> standard processors call context.yield() when they have no work to do.
> I changed my scheduling strategy to CRON Driven with its run schedule every 
> 30 seconds, and the onTrigger() was called every 30 seconds, even after a 
> yield.  So CRON Driven scheduling is working as expected after a yield.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-1452) Yield Duration can short circuit long Timer Driven Run Schedule

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023392#comment-16023392
 ] 

ASF subversion and git services commented on NIFI-1452:
---

Commit 232380dbfd59de45c4c6623f141d6e7052c367f9 in nifi's branch 
refs/heads/master from [~boardm26]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=232380d ]

NIFI-1452 on timer-driven yield, use the greater of yield duration or run 
schedule

This closes #1832.


> Yield Duration can short circuit long Timer Driven Run Schedule
> ---
>
> Key: NIFI-1452
> URL: https://issues.apache.org/jira/browse/NIFI-1452
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.4.1
>Reporter: Michael Moser
>Assignee: Michael Moser
>Priority: Trivial
>
> This may be a rare use case, but I configured a GetFile processor to be Timer 
> Driven with a Run Schedule of 30 secs.  Its Yield Duration was default 1 sec. 
>  I expected GetFile onTrigger() to be called every 30 secs, but it was being 
> called every 1 sec most of the time.
> GetFile will call context.yield() when it polls a directory and gets an empty 
> list in return.  It appears that a yield will ignore the Run Schedule.  Many 
> standard processors call context.yield() when they have no work to do.
> I changed my scheduling strategy to CRON Driven with its run schedule every 
> 30 seconds, and the onTrigger() was called every 30 seconds, even after a 
> yield.  So CRON Driven scheduling is working as expected after a yield.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1852: NIFI-3671: Ensure that we use the existing Resource...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1852


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3671) GenerateFlowFile throws NPE in FileSystemRepository

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023389#comment-16023389
 ] 

ASF GitHub Bot commented on NIFI-3671:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1852


> GenerateFlowFile throws NPE in FileSystemRepository
> ---
>
> Key: NIFI-3671
> URL: https://issues.apache.org/jira/browse/NIFI-3671
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
> Environment: Ubuntu Linux 14.04.5 LTS, x86, NiFi 1.2.0-SNAPSHOT, Java 
> 1.8.0_102
>Reporter: Joe Skora
>Assignee: Mark Payne
> Fix For: 1.3.0
>
> Attachments: flow.xml, genff-npe.log, priority.groovy
>
>
> While volume testing, {{GenerateFlowFile}} is randomly throwing 
> {{NullPointerException}} during startup.  The NPE occurs in 
> {{FileSystemRepository.write()}} called from 
> {{BufferedOutputStream.flushBuffer()}} and results in an Administrative Yield.
> When it happens, the bulletin shows up on the processors immediately after 
> starting them.  Starting them a second time will often succeed, but sometimes 
> it takes a multiple retries.  When the {{GenerateFlowFile}} processors are 
> started for these tests, the graph is typically empty.
> Log entries for an example Thread are attached.  This looks similar to 
> [NIFI-2551|https://issues.apache.org/jira/browse/NIFI-2551], but that is 
> marked as being fixed in 1.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3671) GenerateFlowFile throws NPE in FileSystemRepository

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023387#comment-16023387
 ] 

ASF subversion and git services commented on NIFI-3671:
---

Commit b12cf8a6d28c2a025831fff9363adf2b13199071 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b12cf8a ]

NIFI-3671: This closes #1852. Ensure that we use the existing ResourceClaim (if 
it exists) when swapping data in, instead of creating a new one. Otherwise, if 
the ResourceClaim is still writable, then we may archive the data and then 
write to it, which can cause a NullPointerException in FileSystemRepository

Signed-off-by: joewitt 


> GenerateFlowFile throws NPE in FileSystemRepository
> ---
>
> Key: NIFI-3671
> URL: https://issues.apache.org/jira/browse/NIFI-3671
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
> Environment: Ubuntu Linux 14.04.5 LTS, x86, NiFi 1.2.0-SNAPSHOT, Java 
> 1.8.0_102
>Reporter: Joe Skora
>Assignee: Mark Payne
> Fix For: 1.3.0
>
> Attachments: flow.xml, genff-npe.log, priority.groovy
>
>
> While volume testing, {{GenerateFlowFile}} is randomly throwing 
> {{NullPointerException}} during startup.  The NPE occurs in 
> {{FileSystemRepository.write()}} called from 
> {{BufferedOutputStream.flushBuffer()}} and results in an Administrative Yield.
> When it happens, the bulletin shows up on the processors immediately after 
> starting them.  Starting them a second time will often succeed, but sometimes 
> it takes a multiple retries.  When the {{GenerateFlowFile}} processors are 
> started for these tests, the graph is typically empty.
> Log entries for an example Thread are attached.  This looks similar to 
> [NIFI-2551|https://issues.apache.org/jira/browse/NIFI-2551], but that is 
> marked as being fixed in 1.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3671) GenerateFlowFile throws NPE in FileSystemRepository

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-3671:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1

built/tested described scenario/verified.  Looks good merged to master.  thanks

> GenerateFlowFile throws NPE in FileSystemRepository
> ---
>
> Key: NIFI-3671
> URL: https://issues.apache.org/jira/browse/NIFI-3671
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
> Environment: Ubuntu Linux 14.04.5 LTS, x86, NiFi 1.2.0-SNAPSHOT, Java 
> 1.8.0_102
>Reporter: Joe Skora
>Assignee: Mark Payne
> Fix For: 1.3.0
>
> Attachments: flow.xml, genff-npe.log, priority.groovy
>
>
> While volume testing, {{GenerateFlowFile}} is randomly throwing 
> {{NullPointerException}} during startup.  The NPE occurs in 
> {{FileSystemRepository.write()}} called from 
> {{BufferedOutputStream.flushBuffer()}} and results in an Administrative Yield.
> When it happens, the bulletin shows up on the processors immediately after 
> starting them.  Starting them a second time will often succeed, but sometimes 
> it takes a multiple retries.  When the {{GenerateFlowFile}} processors are 
> started for these tests, the graph is typically empty.
> Log entries for an example Thread are attached.  This looks similar to 
> [NIFI-2551|https://issues.apache.org/jira/browse/NIFI-2551], but that is 
> marked as being fixed in 1.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-1452) Yield Duration can short circuit long Timer Driven Run Schedule

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023380#comment-16023380
 ] 

ASF GitHub Bot commented on NIFI-1452:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1832
  
@joewitt @mosermw the change looks good to me, too. Thanks!


> Yield Duration can short circuit long Timer Driven Run Schedule
> ---
>
> Key: NIFI-1452
> URL: https://issues.apache.org/jira/browse/NIFI-1452
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.4.1
>Reporter: Michael Moser
>Assignee: Michael Moser
>Priority: Trivial
>
> This may be a rare use case, but I configured a GetFile processor to be Timer 
> Driven with a Run Schedule of 30 secs.  Its Yield Duration was default 1 sec. 
>  I expected GetFile onTrigger() to be called every 30 secs, but it was being 
> called every 1 sec most of the time.
> GetFile will call context.yield() when it polls a directory and gets an empty 
> list in return.  It appears that a yield will ignore the Run Schedule.  Many 
> standard processors call context.yield() when they have no work to do.
> I changed my scheduling strategy to CRON Driven with its run schedule every 
> 30 seconds, and the onTrigger() was called every 30 seconds, even after a 
> yield.  So CRON Driven scheduling is working as expected after a yield.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1832: NIFI-1452 on timer-driven yield, use the greater of yield ...

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1832
  
@joewitt @mosermw the change looks good to me, too. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3964) Grok processors - Allow multiple pattern configuration files

[Pierre Villard (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023375#comment-16023375
 ] 

Pierre Villard commented on NIFI-3964:
--

I had ExtractGrok and GrokReader in mind, they both expose a property to link a 
pattern file to load.

> Grok processors - Allow multiple pattern configuration files
> 
>
> Key: NIFI-3964
> URL: https://issues.apache.org/jira/browse/NIFI-3964
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
>
> When using a lot of grok processors with different teams and use cases, it 
> would be useful to be able to split the pattern file in different files that 
> can be individually managed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3964) Grok processors - Allow multiple pattern configuration files

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023363#comment-16023363
 ] 

Joseph Witt commented on NIFI-3964:
---

i think i assumed you were talking about the GrokReader controller service but 
that might have been wrong.  Can you describe which specific processor(s) and 
controller service(s) you had in mind?  I could be completely off base on this 
:)

> Grok processors - Allow multiple pattern configuration files
> 
>
> Key: NIFI-3964
> URL: https://issues.apache.org/jira/browse/NIFI-3964
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
>
> When using a lot of grok processors with different teams and use cases, it 
> would be useful to be able to split the pattern file in different files that 
> can be individually managed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3964) Grok processors - Allow multiple pattern configuration files

[Pierre Villard (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023351#comment-16023351
 ] 

Pierre Villard commented on NIFI-3964:
--

Hey [~joewitt], I agree that we should avoid the need to deploy files over the 
NiFi nodes. I'm not sure to understand the Lookup Service approach though. All 
the defined patterns need to be loaded in the Grok instance when the processor 
is scheduled. Can't see how the lookup api can provide a "get all" patterns 
that would be defined in the controller service. Did you have in mind to add a 
method in the lookup service interface to get all key/value pairs defined in 
the controller service and add a controller service where key/value pairs are 
defined using dynamic properties?

> Grok processors - Allow multiple pattern configuration files
> 
>
> Key: NIFI-3964
> URL: https://issues.apache.org/jira/browse/NIFI-3964
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
>
> When using a lot of grok processors with different teams and use cases, it 
> would be useful to be able to split the pattern file in different files that 
> can be individually managed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3671) GenerateFlowFile throws NPE in FileSystemRepository

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023329#comment-16023329
 ] 

ASF GitHub Bot commented on NIFI-3671:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/1852
  
reviewing


> GenerateFlowFile throws NPE in FileSystemRepository
> ---
>
> Key: NIFI-3671
> URL: https://issues.apache.org/jira/browse/NIFI-3671
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
> Environment: Ubuntu Linux 14.04.5 LTS, x86, NiFi 1.2.0-SNAPSHOT, Java 
> 1.8.0_102
>Reporter: Joe Skora
>Assignee: Mark Payne
> Fix For: 1.3.0
>
> Attachments: flow.xml, genff-npe.log, priority.groovy
>
>
> While volume testing, {{GenerateFlowFile}} is randomly throwing 
> {{NullPointerException}} during startup.  The NPE occurs in 
> {{FileSystemRepository.write()}} called from 
> {{BufferedOutputStream.flushBuffer()}} and results in an Administrative Yield.
> When it happens, the bulletin shows up on the processors immediately after 
> starting them.  Starting them a second time will often succeed, but sometimes 
> it takes a multiple retries.  When the {{GenerateFlowFile}} processors are 
> started for these tests, the graph is typically empty.
> Log entries for an example Thread are attached.  This looks similar to 
> [NIFI-2551|https://issues.apache.org/jira/browse/NIFI-2551], but that is 
> marked as being fixed in 1.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3644) Add DetectDuplicateUsingHBase processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023328#comment-16023328
 ] 

ASF GitHub Bot commented on NIFI-3644:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1645
  
Sorry for taking so long to get back to this...

I tested this using PutDistributedMapCache and FetchDistributedMapCache, 
and noticed the value coming back from fetch wasn't exactly what I had stored. 

In HBaseRowHandler we had:
`lastResultBytes = resultCell.getValueArray()`

And we need:
`lastResultBytes = Arrays.copyOfRange(resultCell.getValueArray(), 
resultCell.getValueOffset(), resultCell.getValueLength() + 
resultCell.getValueOffset());
`

I made a commit here that includes the change:

https://github.com/bbende/nifi/commit/dc8f14d95d6cdbab2aa6e815269fe0d98faa2fe6

I also moved MockHBaseClientService into it's own class so it can be used 
by both tests, so that we don't have to duplicate that code.

Everything else looks good so I will go ahead and merge these changes 
together (your commit then mine). 

Thanks again for contributing! and sorry for the delay.



> Add DetectDuplicateUsingHBase processor
> ---
>
> Key: NIFI-3644
> URL: https://issues.apache.org/jira/browse/NIFI-3644
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Bjorn Olsen
>Priority: Minor
>
> The DetectDuplicate processor makes use of a distributed map cache for 
> maintaining a list of unique file identifiers (such as hashes).
> The distributed map cache functionality could be provided by an HBase table, 
> which then allows for reliably storing a huge volume of file identifiers and 
> auditing information. The downside of this approach is of course that HBase 
> is required.
> Storing the unique file identifiers in a reliable, query-able manner along 
> with some audit information is of benefit to several use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1852: NIFI-3671: Ensure that we use the existing ResourceClaim (...

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/1852
  
reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1645: NIFI-3644 - Added HBase_1_1_2_ClientMapCacheService

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1645
  
Sorry for taking so long to get back to this...

I tested this using PutDistributedMapCache and FetchDistributedMapCache, 
and noticed the value coming back from fetch wasn't exactly what I had stored. 

In HBaseRowHandler we had:
`lastResultBytes = resultCell.getValueArray()`

And we need:
`lastResultBytes = Arrays.copyOfRange(resultCell.getValueArray(), 
resultCell.getValueOffset(), resultCell.getValueLength() + 
resultCell.getValueOffset());
`

I made a commit here that includes the change:

https://github.com/bbende/nifi/commit/dc8f14d95d6cdbab2aa6e815269fe0d98faa2fe6

I also moved MockHBaseClientService into it's own class so it can be used 
by both tests, so that we don't have to duplicate that code.

Everything else looks good so I will go ahead and merge these changes 
together (your commit then mine). 

Thanks again for contributing! and sorry for the delay.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3671) GenerateFlowFile throws NPE in FileSystemRepository

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023322#comment-16023322
 ] 

Mark Payne commented on NIFI-3671:
--

Okay, so I was able to replicate this quite easily now, in a very slightly 
different fashion than you described. I created the 3 GenerateFlowFile procs 
like you mentioned and sent all to UpdateAttribute. I then start the 3 
processors and then stop them. All works fine the first time. I then start 
UpdateAttribute, which swaps the data back in and destroys it. Then when the 
queues are empty, I stopped UpdateAttribute and started GenerateFlowFile again, 
which causes the NPE. It is due to a bug in swapping the data in. Attached is a 
PR that addresses it. Should be very easy to replicate & verify fix.

> GenerateFlowFile throws NPE in FileSystemRepository
> ---
>
> Key: NIFI-3671
> URL: https://issues.apache.org/jira/browse/NIFI-3671
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
> Environment: Ubuntu Linux 14.04.5 LTS, x86, NiFi 1.2.0-SNAPSHOT, Java 
> 1.8.0_102
>Reporter: Joe Skora
>Assignee: Mark Payne
> Fix For: 1.3.0
>
> Attachments: flow.xml, genff-npe.log, priority.groovy
>
>
> While volume testing, {{GenerateFlowFile}} is randomly throwing 
> {{NullPointerException}} during startup.  The NPE occurs in 
> {{FileSystemRepository.write()}} called from 
> {{BufferedOutputStream.flushBuffer()}} and results in an Administrative Yield.
> When it happens, the bulletin shows up on the processors immediately after 
> starting them.  Starting them a second time will often succeed, but sometimes 
> it takes a multiple retries.  When the {{GenerateFlowFile}} processors are 
> started for these tests, the graph is typically empty.
> Log entries for an example Thread are attached.  This looks similar to 
> [NIFI-2551|https://issues.apache.org/jira/browse/NIFI-2551], but that is 
> marked as being fixed in 1.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)