[GitHub] [nifi] exceptionfactory commented on pull request #5073: NIFI-8538 Upgraded Apache Commons IO to 2.8.0

[GitBox]

exceptionfactory commented on pull request #5073:
URL: https://github.com/apache/nifi/pull/5073#issuecomment-840250808


   PR build on Windows timed out but local repository build completed:
   
   https://github.com/exceptionfactory/nifi/actions/runs/837405438


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (NIFI-8461) Queue reports items but cannot list them

[KevinSky (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343648#comment-17343648
 ] 

KevinSky edited comment on NIFI-8461 at 5/13/21, 1:16 AM:
--

hi Mark Payne, and [#Matt Burgess],I found ExecuteScript::onTrigger():
{code:java}
configurator.init(scriptEngine, scriptToRun, 
scriptingComponentHelper.getModules());
{code}
JythonScriptEngineConfigurator::init()
{code:java}
@Override
public Object init(ScriptEngine engine, String scriptBody, String[] 
modulePaths) throws ScriptException {
// Always compile when first run
if (engine != null && compiledScriptRef.get() == null) {
// Add prefix for import sys and all jython modules
String prefix = "import sys\n"
+ Arrays.stream(modulePaths).map((modulePath) -> 
"sys.path.append(" + PyString.encode_UnicodeEscape(modulePath, true) + ")")
.collect(Collectors.joining("\n"));
final CompiledScript compiled = ((Compilable) 
engine).compile(prefix + scriptBody);
compiledScriptRef.set(compiled);
}
return compiledScriptRef.get();
}
{code}
compiledScriptRef.get() is always not null after executeScript processor 
restart. So I infer whether it is because the compilation is not restarted 
after the stop, which causes the session stored in the bindings in the 
executesScript to be the previous one, so the flowfile is not correctly set to 
ack status.

I found this compiledScriptRef is store in 
scriptingComponentHelper.scriptEngineConfiguratorMap,but executeScript 
processor stop,it only clear the scriptingComponentHelper.engineQ,but 
scriptingComponentHelper.scriptEngineConfiguratorMap also has the reference.so 
I just modify the scriptingComponentHelper::stop()
{code:java}
public void stop() {
if (engineQ != null) {
engineQ.clear();
}
if (!scriptEngineConfiguratorMap.isEmpty()){
scriptEngineConfiguratorMap.clear();
}
}
{code}
It works,but i found invokeScriptedProcessor also use 
scriptingComponentHelper::stop(),I don't know this modification is suitable for 
this processor.


was (Author: kevinsky1093):
hi Mark Payne, and [#Matt Burgess],I found ExecuteScript::onTrigger():
{code:java}
// configurator.init(scriptEngine, scriptToRun, 
scriptingComponentHelper.getModules());
{code}
 

 

 

JythonScriptEngineConfigurator::init()
{code:java}
//@Override
public Object init(ScriptEngine engine, String scriptBody, String[] 
modulePaths) throws ScriptException {
// Always compile when first run
if (engine != null && compiledScriptRef.get() == null) {
// Add prefix for import sys and all jython modules
String prefix = "import sys\n"
+ Arrays.stream(modulePaths).map((modulePath) -> 
"sys.path.append(" + PyString.encode_UnicodeEscape(modulePath, true) + ")")
.collect(Collectors.joining("\n"));
final CompiledScript compiled = ((Compilable) 
engine).compile(prefix + scriptBody);
compiledScriptRef.set(compiled);
}
return compiledScriptRef.get();
}
{code}
compiledScriptRef.get() is always not null after executeScript processor 
restart. So I infer whether it is because the compilation is not restarted 
after the stop, which causes the session stored in the bindings in the 
executesScript to be the previous one, so the flowfile is not correctly set to 
ack status.

I found this compiledScriptRef is store in 
scriptingComponentHelper.scriptEngineConfiguratorMap,but executeScript 
processor stop,it only clear the scriptingComponentHelper.engineQ,but 
scriptingComponentHelper.scriptEngineConfiguratorMap also has the reference.so 
I just modify the scriptingComponentHelper::stop()
{code:java}
// public void stop() {
if (engineQ != null) {
engineQ.clear();
}
if (!scriptEngineConfiguratorMap.isEmpty()){
scriptEngineConfiguratorMap.clear();
}
}
{code}
It works,but i found invokeScriptedProcessor also use 
scriptingComponentHelper::stop(),I don't know this modification is suitable for 
this processor.

> Queue reports items but cannot list them
> 
>
> Key: NIFI-8461
> URL: https://issues.apache.org/jira/browse/NIFI-8461
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.13.2
> Environment: Docker image apache/nifi :1.13.2, Windows 10 Pro, WSL 2 
> (Ubuntu), x86_64
>Reporter: Kevin Aagaard
>Assignee: Matt Burgess
>Priority: Major
>  Labels: queue
> Attachments: ExecuteScript_Queue_Issue.txt, NIFI-8461.zip, 
> image-2021-04-22-08-48-58-117.png, image-2021-04-22-08-49-40-904.png, 
> image-2021-04-27-14-23-07-849.png, image-2021-04-27-14-23-24-477.png
>
>
> The following segment of wor

[jira] [Commented] (NIFI-8461) Queue reports items but cannot list them

[KevinSky (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343648#comment-17343648
 ] 

KevinSky commented on NIFI-8461:


hi Mark Payne, and [#Matt Burgess],I found ExecuteScript::onTrigger():
{code:java}
// configurator.init(scriptEngine, scriptToRun, 
scriptingComponentHelper.getModules());
{code}
 

 

 

JythonScriptEngineConfigurator::init()
{code:java}
//@Override
public Object init(ScriptEngine engine, String scriptBody, String[] 
modulePaths) throws ScriptException {
// Always compile when first run
if (engine != null && compiledScriptRef.get() == null) {
// Add prefix for import sys and all jython modules
String prefix = "import sys\n"
+ Arrays.stream(modulePaths).map((modulePath) -> 
"sys.path.append(" + PyString.encode_UnicodeEscape(modulePath, true) + ")")
.collect(Collectors.joining("\n"));
final CompiledScript compiled = ((Compilable) 
engine).compile(prefix + scriptBody);
compiledScriptRef.set(compiled);
}
return compiledScriptRef.get();
}
{code}
compiledScriptRef.get() is always not null after executeScript processor 
restart. So I infer whether it is because the compilation is not restarted 
after the stop, which causes the session stored in the bindings in the 
executesScript to be the previous one, so the flowfile is not correctly set to 
ack status.

I found this compiledScriptRef is store in 
scriptingComponentHelper.scriptEngineConfiguratorMap,but executeScript 
processor stop,it only clear the scriptingComponentHelper.engineQ,but 
scriptingComponentHelper.scriptEngineConfiguratorMap also has the reference.so 
I just modify the scriptingComponentHelper::stop()
{code:java}
// public void stop() {
if (engineQ != null) {
engineQ.clear();
}
if (!scriptEngineConfiguratorMap.isEmpty()){
scriptEngineConfiguratorMap.clear();
}
}
{code}
It works,but i found invokeScriptedProcessor also use 
scriptingComponentHelper::stop(),I don't know this modification is suitable for 
this processor.

> Queue reports items but cannot list them
> 
>
> Key: NIFI-8461
> URL: https://issues.apache.org/jira/browse/NIFI-8461
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.13.2
> Environment: Docker image apache/nifi :1.13.2, Windows 10 Pro, WSL 2 
> (Ubuntu), x86_64
>Reporter: Kevin Aagaard
>Assignee: Matt Burgess
>Priority: Major
>  Labels: queue
> Attachments: ExecuteScript_Queue_Issue.txt, NIFI-8461.zip, 
> image-2021-04-22-08-48-58-117.png, image-2021-04-22-08-49-40-904.png, 
> image-2021-04-27-14-23-07-849.png, image-2021-04-27-14-23-24-477.png
>
>
> The following segment of workflow demonstrates the issue. The queue reports 
> that there are items within it, but they cannot be listed, even after 
> stopping the consumer (and producer). Since this is in a Docker Container, it 
> is unlikely an OS error.
>  
> !image-2021-04-22-08-48-58-117.png!
>  
> !image-2021-04-22-08-49-40-904.png!
> I do not currently have a simplified example workflow to recreate the error, 
> but can work on it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-registry] mtien-apache commented on pull request #319: NIFIREG-395 - Implemented the ability to import and export versioned flows through the UI

[GitBox]

mtien-apache commented on pull request #319:
URL: https://github.com/apache/nifi-registry/pull/319#issuecomment-840183039


   Pushed commits that addresses all remaining changes and PR feedback. The PR 
is ready for a full review. 
   @sardell Would you review the frontend changes?
   
   Recent changes include:
   
   - Changed the dialog title from 'Download Version' to 'Export Version' and 
refactored the component. @bbende @andrewmlim @moranr Thanks for the feedback 
and agreed 'Export' seems more consistent  
   
   - Changed the 'Export Version' dialog dropdown menu item to "Latest (Version 
#)"
   - Changed the Actions menu hover style to differentiate from the focus style
   - Added frontend tests. Note - With the way Angular and Javascript is 
currently configured in Registry, I had issues writing certain tests for the 
new components. It's been captured in a JIRA ticket: 
[NIFIREG-454](https://issues.apache.org/jira/browse/NIFIREG-454). Once 
resolved, more tests can be added.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-8536) PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port

[Nadeem (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nadeem updated NIFI-8536:
-
Status: Patch Available  (was: Open)

> PrometheusReportingTask and PrometheusRecordSink do not support variable 
> registry for port
> --
>
> Key: NIFI-8536
> URL: https://issues.apache.org/jira/browse/NIFI-8536
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions, Variable Registry
>Affects Versions: 1.13.2, 1.12.1
>Reporter: Matthew Clarke
>Assignee: Nadeem
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The documentation for both PrometheusReportingTask [1] (reporting task) and 
> PrometheusRecordSink [2] (controller service) state that NiFi Variable 
> Registry can be used to define the port. However, configuring NiFi EL in the 
> port property results in:
> java.lang.NumberFormatException: For input string: "${ property name>}" 
> Support variable registry here would allow user to set a unique port on each 
> host in a NiFi cluster.
> [1] 
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html
> [2] 
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusRecordSink/index.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] naddym opened a new pull request #5075: NIFI-8536: PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port property

[GitBox]

naddym opened a new pull request #5075:
URL: https://github.com/apache/nifi/pull/5075


   
   Thank you for submitting a contribution to Apache NiFi.
   
   Please provide a short description of the PR here:
   
    Description of PR
   
   _Enables X functionality; fixes bug NIFI-._
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [x] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [x] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [x] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [x] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [ ] Have you verified that the full build is successful on JDK 8?
   - [ ] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (NIFIREG-454) Unable to write frontend tests

[M Tien (Jira)]

M Tien created NIFIREG-454:
--

 Summary: Unable to write frontend tests
 Key: NIFIREG-454
 URL: https://issues.apache.org/jira/browse/NIFIREG-454
 Project: NiFi Registry
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: M Tien
 Attachments: Screen Shot 2021-04-22 at 10.13.06 AM.png, Screen Shot 
2021-05-12 at 4.21.03 PM.png

With the current frontend configuration of Angular and Javascript, it does not 
allow certain frontend tests to be added. 

*Example 1*

Some issues that have arisen include, services that cannot be injected into 
components. It produces a compile error logged in the console:

!Screen Shot 2021-04-22 at 10.13.06 AM.png!

 

*Example 2*

Unable to create a Jasmine spy:

!Screen Shot 2021-05-12 at 4.21.03 PM.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (NIFI-8536) PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port

[Nadeem (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nadeem reassigned NIFI-8536:


Assignee: Nadeem

> PrometheusReportingTask and PrometheusRecordSink do not support variable 
> registry for port
> --
>
> Key: NIFI-8536
> URL: https://issues.apache.org/jira/browse/NIFI-8536
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions, Variable Registry
>Affects Versions: 1.12.1, 1.13.2
>Reporter: Matthew Clarke
>Assignee: Nadeem
>Priority: Minor
>
> The documentation for both PrometheusReportingTask [1] (reporting task) and 
> PrometheusRecordSink [2] (controller service) state that NiFi Variable 
> Registry can be used to define the port. However, configuring NiFi EL in the 
> port property results in:
> java.lang.NumberFormatException: For input string: "${ property name>}" 
> Support variable registry here would allow user to set a unique port on each 
> host in a NiFi cluster.
> [1] 
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html
> [2] 
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusRecordSink/index.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] thenatog opened a new pull request #5074: NIFI-8343 - Updated solr from 8.4.1 to 8.8.2. Small code changes were…

[GitBox]

thenatog opened a new pull request #5074:
URL: https://github.com/apache/nifi/pull/5074


   … required.
   
   
   Thank you for submitting a contribution to Apache NiFi.
   
   Please provide a short description of the PR here:
   
    Description of PR
   
   _Enables X functionality; fixes bug NIFI-._
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [x] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [x] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [x] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [x] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [x] Have you verified that the full build is successful on JDK 8?
   - [ ] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-registry] mtien-apache commented on a change in pull request #319: NIFIREG-395 - Implemented the ability to import and export versioned flows through the UI

[GitBox]

mtien-apache commented on a change in pull request #319:
URL: https://github.com/apache/nifi-registry/pull/319#discussion_r631410991



##
File path: 
nifi-registry-core/nifi-registry-web-ui/src/main/webapp/components/explorer/grid-list/dialogs/download-versioned-flow/nf-registry-download-versioned-flow.js
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Component } from '@angular/core';
+import NfRegistryApi from 'services/nf-registry.api';
+import { MAT_DIALOG_DATA, MatDialogRef } from '@angular/material';
+import { FdsSnackBarService } from '@nifi-fds/core';
+
+/**
+ * NfRegistryDownloadVersionedFlow constructor.
+ *
+ * @param nfRegistryApi The api service.
+ * @param fdsSnackBarServiceThe FDS snack bar service module.
+ * @param matDialogRef  The angular material dialog ref.
+ * @param data  The data passed into this component.
+ * @constructor
+ */
+function NfRegistryDownloadVersionedFlow(nfRegistryApi, fdsSnackBarService, 
matDialogRef, data) {
+// Services
+this.snackBarService = fdsSnackBarService;
+this.nfRegistryApi = nfRegistryApi;
+this.dialogRef = matDialogRef;
+// local state
+this.keepDialogOpen = false;
+this.protocol = location.protocol;
+this.droplet = data.droplet;
+this.selectedVersion = this.droplet.snapshotMetadata[0].version;
+}
+
+NfRegistryDownloadVersionedFlow.prototype = {
+constructor: NfRegistryDownloadVersionedFlow,
+
+/**
+ * Download specified versioned flow snapshot.
+ */
+downloadVersion: function () {
+var self = this;
+var version = this.selectedVersion;
+
+
this.nfRegistryApi.exportDropletVersionedSnapshot(this.droplet.link.href, 
version).subscribe(function (response) {
+if (!response.status || response.status === 200) {

Review comment:
   @exceptionfactory A successful response has typically been just the 
response body returned. So, when the body is returned (a success) there will 
not be a status property. I can see how this could be confusing at first glance 
and without much context. This if-statement is used throughout the codebase, so 
I decided to follow the conventions.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-registry] mtien-apache commented on a change in pull request #319: NIFIREG-395 - Implemented the ability to import and export versioned flows through the UI

[GitBox]

mtien-apache commented on a change in pull request #319:
URL: https://github.com/apache/nifi-registry/pull/319#discussion_r631404406



##
File path: 
nifi-registry-core/nifi-registry-web-ui/src/main/webapp/services/nf-registry.api.js
##
@@ -75,6 +75,137 @@ NfRegistryApi.prototype = {
 );
 },
 
+/**
+ * Retrieves the specified versioned flow snapshot for an existing droplet 
the registry has stored.
+ *
+ * @param {string}  dropletUri  The uri of the droplet to request.
+ * @param {number}  versionNumber   The version of the flow to request.
+ * @returns {*}
+ */
+exportDropletVersionedSnapshot: function (dropletUri, versionNumber) {
+var self = this;
+var url = '../nifi-registry-api/' + dropletUri + '/versions/' + 
versionNumber + '/export';
+var options = {
+headers: headers,
+observe: 'response'
+};
+
+return this.http.get(url, options).pipe(
+map(function (response) {
+// export the VersionedFlowSnapshot by creating a hidden 
anchor element
+var stringSnapshot = 
encodeURIComponent(JSON.stringify(response.body, null, 2));

Review comment:
   @exceptionfactory Good catch! I can return a `responseType: 'text'` and 
omit calling `JSON.stringify()`. The file content will not be in pretty print 
format but that seems ok to me.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-8538) Upgrade Apache Commons IO to 2.8.0

[David Handermann (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-8538:
---
Status: Patch Available  (was: Open)

> Upgrade Apache Commons IO to 2.8.0
> --
>
> Key: NIFI-8538
> URL: https://issues.apache.org/jira/browse/NIFI-8538
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.13.2
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>  Labels: security
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Apache Commons IO version 2.7 and below are vulnerable to 
> [CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425].  Although 
> NiFi does not appear to have any direct calls to 
> {{FileNameUtils.normalize()}}, numerous libraries leverage Commons IO.  
> Upgrading to version 2.8.0 addresses this issue and also includes a number of 
> other minor bug fixes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-8538) Upgrade Apache Commons IO to 2.8.0

[David Handermann (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-8538:
---
Description: Apache Commons IO version 2.6 and below are vulnerable to 
[CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425]. Although NiFi 
does not appear to have any direct calls to {{FileNameUtils.normalize()}}, 
numerous libraries leverage Commons IO. Upgrading to version 2.8.0 addresses 
this issue and also includes a number of other minor bug fixes.  (was: Apache 
Commons IO version 2.7 and below are vulnerable to 
[CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425].  Although 
NiFi does not appear to have any direct calls to {{FileNameUtils.normalize()}}, 
numerous libraries leverage Commons IO.  Upgrading to version 2.8.0 addresses 
this issue and also includes a number of other minor bug fixes.)

> Upgrade Apache Commons IO to 2.8.0
> --
>
> Key: NIFI-8538
> URL: https://issues.apache.org/jira/browse/NIFI-8538
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.13.2
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>  Labels: security
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Apache Commons IO version 2.6 and below are vulnerable to 
> [CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425]. Although 
> NiFi does not appear to have any direct calls to 
> {{FileNameUtils.normalize()}}, numerous libraries leverage Commons IO. 
> Upgrading to version 2.8.0 addresses this issue and also includes a number of 
> other minor bug fixes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] exceptionfactory opened a new pull request #5073: NIFI-8538 Upgraded Apache Commons IO to 2.8.0

[GitBox]

exceptionfactory opened a new pull request #5073:
URL: https://github.com/apache/nifi/pull/5073


    Description of PR
   
   NIFI-8538 Upgrades Apache Commons IO to 2.8.0 in both direct and transitive 
dependencies to address 
[CVE-2021-29425](https://nvd.nist.gov/vuln/detail/CVE-2021-29425) and 
incorporate other minor bug fixes.  Most direct dependencies involved an 
upgrade from 2.6 while some transitive dependencies required custom dependency 
management configuration changes to upgrade from 2.4 and 2.5.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [X] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [X] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [X] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [X] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [X] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [ ] Have you verified that the full build is successful on JDK 8?
   - [X] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (NIFI-8538) Upgrade Apache Commons IO to 2.8.0

[David Handermann (Jira)]

David Handermann created NIFI-8538:
--

 Summary: Upgrade Apache Commons IO to 2.8.0
 Key: NIFI-8538
 URL: https://issues.apache.org/jira/browse/NIFI-8538
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Affects Versions: 1.13.2
Reporter: David Handermann
Assignee: David Handermann


Apache Commons IO version 2.7 and below are vulnerable to 
[CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425].  Although 
NiFi does not appear to have any direct calls to {{FileNameUtils.normalize()}}, 
numerous libraries leverage Commons IO.  Upgrading to version 2.8.0 addresses 
this issue and also includes a number of other minor bug fixes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] gresockj opened a new pull request #5072: WIP NIFI-8490: Adding inherited parameter contexts

[GitBox]

gresockj opened a new pull request #5072:
URL: https://github.com/apache/nifi/pull/5072


   
   
    Description of PR
   
   Adds "Inherited Parameter Contexts" to ParameterContexts.
   
   Although these inherited parameter contexts should function as expected in 
the UI, the UI piece of setting the inherited contexts is not implemented yet.  
Therefore the easiest way to test out the functionality is through the CLI:
   
   ```
   #> nifi set-inherited-param-contexts -pcid  -pcii 

   ```
   
   Also, in order to build the branch before nifi-registry is pulled into nifi, 
you will need to add the following to 
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/flow/VersionedParameterContext.java
 in the nifi-registry project:
   
   ```
   private List parameterContexts;
   
   @ApiModelProperty("The names of additional parameter contexts from which 
to inherit parameters")
   public List getParameterContexts() {
   return parameterContexts;
   }
   
   public void setParameterContexts(List parameterContextNames) {
   this.parameterContexts = parameterContextNames;
   }
   ```
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [x] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [x] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [x] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [ ] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [ ] Have you verified that the full build is successful on JDK 8?
   - [ ] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] mattyb149 commented on a change in pull request #5070: NIFI-8535: Better error message in PutDatabaseRecord when table does not exist

[GitBox]

mattyb149 commented on a change in pull request #5070:
URL: https://github.com/apache/nifi/pull/5070#discussion_r631343009



##
File path: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/processors/standard/TestPutDatabaseRecord.groovy
##
@@ -483,6 +483,9 @@ class TestPutDatabaseRecord {
 
 runner.assertTransferCount(PutDatabaseRecord.REL_SUCCESS, 0)
 runner.assertTransferCount(PutDatabaseRecord.REL_FAILURE, 1)
+MockFlowFile flowFile = 
runner.getFlowFilesForRelationship(PutDatabaseRecord.REL_FAILURE).get(0);
+flowFile.assertAttributeEquals('putdatabaserecord.error',
+'java.sql.SQLException: Table PERSONS2 not found, ensure the 
Catalog, Schema, and/or Table Names match those in the database exactly')

Review comment:
   Good point, will change




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-8537) Components become invalid if referencing a disabled controller service, even if referencing property is not in scope due to dependencies

[Mark Payne (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-8537:
-
Status: Patch Available  (was: Open)

> Components become invalid if referencing a disabled controller service, even 
> if referencing property is not in scope due to dependencies
> 
>
> Key: NIFI-8537
> URL: https://issues.apache.org/jira/browse/NIFI-8537
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> If a processor/controller service property (A) has a dependency on another 
> property (B), we do not perform validation of property A (i.e., the property 
> descriptor's validator is ignored). However, if Property A references a 
> Controller Service, and that Controller Service is either disabled or 
> invalid, we currently consider the referencing component invalid. This should 
> not be the case.
> To replicate:
>  * Create an AvroSchemaRegistry Controller Service. Leave the Controller 
> Service disabled.
>  * Create a JsonTreeReader controller service.
>  * Configure the JsonTreeReader Controller Service.
>  * Set "Schema Access Strategy" to "Use 'Schema Name' Property"
>  * Set "Schema Registry" to the AvroSchemaRegistry created in the first step.
>  * Change "Schema Access Strategy" to "Infer Schema"
>  * Click Apply to apply the changes.
> The JsonTreeReader service will be invalid due to the fact that 
> AvroSchemaRegistry is disabled. Not only should the Schema Registry property 
> not be considered in validation, it becomes very confusing now because the 
> validation error states: "'Schema Registry' validated against XYZ is invalid 
> because Controller Service with ID XYZ is invalid." But when configuring 
> JsonTreeReader, we don't even see the "Schema Registry" property.
> The reference should not be validated unless its property's dependencies have 
> been satisfied.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] markap14 opened a new pull request #5071: NIFI-8537: Check that a property's dependencies are satisfied before …

[GitBox]

markap14 opened a new pull request #5071:
URL: https://github.com/apache/nifi/pull/5071


   …checking if the referenced controller service valid/enabled
   
   
   Thank you for submitting a contribution to Apache NiFi.
   
   Please provide a short description of the PR here:
   
    Description of PR
   
   _Enables X functionality; fixes bug NIFI-._
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [ ] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [ ] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [ ] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [ ] Have you verified that the full build is successful on JDK 8?
   - [ ] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] markap14 commented on a change in pull request #5070: NIFI-8535: Better error message in PutDatabaseRecord when table does not exist

[GitBox]

markap14 commented on a change in pull request #5070:
URL: https://github.com/apache/nifi/pull/5070#discussion_r631331669



##
File path: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/processors/standard/TestPutDatabaseRecord.groovy
##
@@ -483,6 +483,9 @@ class TestPutDatabaseRecord {
 
 runner.assertTransferCount(PutDatabaseRecord.REL_SUCCESS, 0)
 runner.assertTransferCount(PutDatabaseRecord.REL_FAILURE, 1)
+MockFlowFile flowFile = 
runner.getFlowFilesForRelationship(PutDatabaseRecord.REL_FAILURE).get(0);
+flowFile.assertAttributeEquals('putdatabaserecord.error',
+'java.sql.SQLException: Table PERSONS2 not found, ensure the 
Catalog, Schema, and/or Table Names match those in the database exactly')

Review comment:
   The SQLException that gets thrown here turns into an attribute. But the 
text of that Exception is intended for human consumption and can be freely 
changed at any time. We should not see test failures just because the text of 
an Exception was changed. Probably makes sense to instead make more specific 
assertions about the text such as:
   ```
   final String errorMessage = flowFile.getAttribute("putdatabaserecord.error");
   assertTrue(errorMessage.contains("PERSONS2");
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (NIFI-8537) Components become invalid if referencing a disabled controller service, even if referencing property is not in scope due to dependencies

[Mark Payne (Jira)]

Mark Payne created NIFI-8537:


 Summary: Components become invalid if referencing a disabled 
controller service, even if referencing property is not in scope due to 
dependencies
 Key: NIFI-8537
 URL: https://issues.apache.org/jira/browse/NIFI-8537
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne


If a processor/controller service property (A) has a dependency on another 
property (B), we do not perform validation of property A (i.e., the property 
descriptor's validator is ignored). However, if Property A references a 
Controller Service, and that Controller Service is either disabled or invalid, 
we currently consider the referencing component invalid. This should not be the 
case.

To replicate:
 * Create an AvroSchemaRegistry Controller Service. Leave the Controller 
Service disabled.
 * Create a JsonTreeReader controller service.
 * Configure the JsonTreeReader Controller Service.
 * Set "Schema Access Strategy" to "Use 'Schema Name' Property"
 * Set "Schema Registry" to the AvroSchemaRegistry created in the first step.
 * Change "Schema Access Strategy" to "Infer Schema"
 * Click Apply to apply the changes.

The JsonTreeReader service will be invalid due to the fact that 
AvroSchemaRegistry is disabled. Not only should the Schema Registry property 
not be considered in validation, it becomes very confusing now because the 
validation error states: "'Schema Registry' validated against XYZ is invalid 
because Controller Service with ID XYZ is invalid." But when configuring 
JsonTreeReader, we don't even see the "Schema Registry" property.

The reference should not be validated unless its property's dependencies have 
been satisfied.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] mattyb149 opened a new pull request #5070: NIFI-8535: Better error message in PutDatabaseRecord when table does not exist

[GitBox]

mattyb149 opened a new pull request #5070:
URL: https://github.com/apache/nifi/pull/5070


   
   Thank you for submitting a contribution to Apache NiFi.
   
   Please provide a short description of the PR here:
   
    Description of PR
   
   If no columns are found for the specified table, call getTables() to see if 
the table exists at all, and return a more descriptive error message in that 
case. Otherwise, warn the user that the specified table has no columns
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [x] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [x] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [x] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [ ] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [x] Have you written or updated unit tests to verify your changes?
   - [x] Have you verified that the full build is successful on JDK 8?
   - [ ] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (NIFI-8536) PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port

[Matthew Clarke (Jira)]

Matthew Clarke created NIFI-8536:


 Summary: PrometheusReportingTask and PrometheusRecordSink do not 
support variable registry for port
 Key: NIFI-8536
 URL: https://issues.apache.org/jira/browse/NIFI-8536
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions, Variable Registry
Affects Versions: 1.13.2, 1.12.1
Reporter: Matthew Clarke


The documentation for both PrometheusReportingTask [1] (reporting task) and 
PrometheusRecordSink [2] (controller service) state that NiFi Variable Registry 
can be used to define the port. However, configuring NiFi EL in the port 
property results in:

java.lang.NumberFormatException: For input string: "${}" 

Support variable registry here would allow user to set a unique port on each 
host in a NiFi cluster.

[1] 
http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html

[2] 
http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.13.2/org.apache.nifi.reporting.prometheus.PrometheusRecordSink/index.html




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (NIFI-8535) PutDatabaseRecord should give a better message when the table cannot be found

[Matt Burgess (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess reassigned NIFI-8535:
--

Assignee: Matt Burgess

> PutDatabaseRecord should give a better message when the table cannot be found
> -
>
> Key: NIFI-8535
> URL: https://issues.apache.org/jira/browse/NIFI-8535
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Currently PutDatabaseRecord calls DatabaseMetaData.getColumns() to try and 
> match the columns from the specified table to the fields in the incoming 
> record(s). However if the table itself is not found, this method returns an 
> empty ResultSet, so it is not known whether the table does not exist or if it 
> exists with no columns.
> PutDatabaseRecord should call DatabaseMetaData.getTables() if the column list 
> is empty, and give a more descriptive error message if the table is not 
> found. This can help the user determine whether there is a field/column 
> mismatch or a catalog/schema/table name mismatch.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-8535) PutDatabaseRecord should give a better message when the table cannot be found

[Matt Burgess (Jira)]

Matt Burgess created NIFI-8535:
--

 Summary: PutDatabaseRecord should give a better message when the 
table cannot be found
 Key: NIFI-8535
 URL: https://issues.apache.org/jira/browse/NIFI-8535
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Matt Burgess


Currently PutDatabaseRecord calls DatabaseMetaData.getColumns() to try and 
match the columns from the specified table to the fields in the incoming 
record(s). However if the table itself is not found, this method returns an 
empty ResultSet, so it is not known whether the table does not exist or if it 
exists with no columns.

PutDatabaseRecord should call DatabaseMetaData.getTables() if the column list 
is empty, and give a more descriptive error message if the table is not found. 
This can help the user determine whether there is a field/column mismatch or a 
catalog/schema/table name mismatch.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-2892) Implement AWS Kinesis Stream Get Processor

[Peter Turcsanyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Turcsanyi updated NIFI-2892:
--
Fix Version/s: 1.14.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Implement AWS Kinesis Stream Get Processor
> --
>
> Key: NIFI-2892
> URL: https://issues.apache.org/jira/browse/NIFI-2892
> Project: Apache NiFi
>  Issue Type: New Feature
>Affects Versions: 1.0.0
>Reporter: Stephane Maarek
>Assignee: Chris Sampson
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: NIFI-2892.xml, NiFi-2892.json
>
>  Time Spent: 15h 40m
>  Remaining Estimate: 0h
>
> As the Kinesis Put Processor was just implemented in #1540, it would be great 
> to have a Kinesis Get Processor.
> The main challenges are making sure the SDKs we use for this have a license 
> that authorise NiFi to bundle them with the main binaries. 
> I also know there are two models to read from a Kinesis Stream (push vs 
> pull), and we may want to implement either or both, as part of one or 
> multiple processors, for performance purposes. Open for discussions



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-2892) Implement AWS Kinesis Stream Get Processor

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343437#comment-17343437
 ] 

ASF subversion and git services commented on NIFI-2892:
---

Commit a274c12c225ab976daff26f8bc67a6fe7614 in nifi's branch 
refs/heads/main from Chris Sampson
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a274c12 ]

NIFI-2892 Implement AWS Kinesis Stream Consume Processor

This closes #4822.

Co-authored-by: uday 

Signed-off-by: Peter Turcsanyi 


> Implement AWS Kinesis Stream Get Processor
> --
>
> Key: NIFI-2892
> URL: https://issues.apache.org/jira/browse/NIFI-2892
> Project: Apache NiFi
>  Issue Type: New Feature
>Affects Versions: 1.0.0
>Reporter: Stephane Maarek
>Assignee: Chris Sampson
>Priority: Major
> Attachments: NIFI-2892.xml, NiFi-2892.json
>
>  Time Spent: 15.5h
>  Remaining Estimate: 0h
>
> As the Kinesis Put Processor was just implemented in #1540, it would be great 
> to have a Kinesis Get Processor.
> The main challenges are making sure the SDKs we use for this have a license 
> that authorise NiFi to bundle them with the main binaries. 
> I also know there are two models to read from a Kinesis Stream (push vs 
> pull), and we may want to implement either or both, as part of one or 
> multiple processors, for performance purposes. Open for discussions



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] asfgit closed pull request #4822: NIFI-2892 Implement AWS Kinesis Stream Get Processor

[GitBox]

asfgit closed pull request #4822:
URL: https://github.com/apache/nifi/pull/4822


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] markobean commented on pull request #5064: NIFI-8007: Setup background grid

[GitBox]

markobean commented on pull request #5064:
URL: https://github.com/apache/nifi/pull/5064#issuecomment-839980571


   Looks great. Thanks for adding the major/minor option.
   
   Here's some testing I did:
   From context menu, selected each of the Background grid options:
 Show only major ticks
 Hide grid
 Show full grid
   Verified each display option was accurate
   Verified only the two options not the current mode displayed, e.g. when Hide 
grid is selected, only Show full grid and Show only major ticks are available 
menu options
   Refreshed display in each mode ensuring grid settings remained the same 
(context menu refresh and browser refresh)
   Tested in Chrome (88.0) and Firefox (60.0) - I know, pretty old, but I can 
retest on a different platform if desired
   Verified settings on one browser did not affect view in another browser. 
(Different tabs of the same browser would update settings when using the 
browser refresh button.)
   Grid scales along with other components on the graph when zooming in and 
out. (This is new behavior, and more natural IMO.)
   
   Overall, LGTM, +1
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] markap14 commented on pull request #5059: NIFI-8519 Adding HDFS support for NAR autoload

[GitBox]

markap14 commented on pull request #5059:
URL: https://github.com/apache/nifi/pull/5059#issuecomment-839940334


   @ottobackwards not sure VFS would provide what we need here. Not sure how it 
deals with all of the authentication etc. But most importantly, it doesn't 
really provide us the pluggability that we'd like to have here. We don't want 
to say "you can put extensions in your local file system or in HDFS." We want 
this to allow pluggability for any number of locations. We would also want to 
account for the fact that some systems may be offline, etc. and so we want to 
download the files locally, not just pull them on-demand each time. And we'd 
also have to consider all of the concerns with classloader isolation. The NAR 
Loader stuff at present is in the root lib. We certainly cannot put any sort of 
Hadoop libs, etc. at the root, so the logic needs to be isolated into NARs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (MINIFICPP-1557) C2VerifyHeartbeatAndStop transient failure

[Gabor Gyimesi (Jira)]

Gabor Gyimesi created MINIFICPP-1557:


 Summary: C2VerifyHeartbeatAndStop transient failure
 Key: MINIFICPP-1557
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1557
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Reporter: Gabor Gyimesi
 Attachments: C2VerifyHeartbeatAndStop_failure_ubuntu20.04.txt





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1070: MINIFICPP-1556 Fix setting partial or invalid credentials in AWS credentials service

[GitBox]

lordgamez commented on a change in pull request #1070:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1070#discussion_r631092918



##
File path: libminifi/test/aws-tests/PutS3ObjectTests.cpp
##
@@ -107,6 +107,17 @@ TEST_CASE_METHOD(PutS3ObjectTestsFixture, "Test required 
property not set", "[aw
   REQUIRE_THROWS_AS(test_controller.runSession(plan, true), 
minifi::Exception&);
 }
 
+TEST_CASE_METHOD(PutS3ObjectTestsFixture, "Test incomplete credentials in 
credentials service", "[awsS3Config]") {
+  setBucket();
+  plan->setProperty(aws_credentials_service, "Secret Key", "secret");
+  setCredentialsService();
+  REQUIRE_THROWS_AS(test_controller.runSession(plan, true), 
minifi::Exception&);
+  REQUIRE(verifyLogLinePresenceInPollTime(std::chrono::seconds(3), "AWS 
Credentials have not been set!"));
+
+  // Test that no invalid credentials file was set from previous properties
+  REQUIRE(!LogTestController::getInstance().contains("load configure file 
failed", std::chrono::seconds(0), std::chrono::milliseconds(0)));

Review comment:
   Updated in 91ec4bf4a04a146344ceab44830f375ed6278338




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-8534) ConvertExcelToCSVProcessor outputting 0-byte files

[John Wise (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Wise updated NIFI-8534:

Affects Version/s: 1.12.1

> ConvertExcelToCSVProcessor outputting 0-byte files
> --
>
> Key: NIFI-8534
> URL: https://issues.apache.org/jira/browse/NIFI-8534
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.12.1
>Reporter: John Wise
>Priority: Critical
>
> We're converting a cache of legacy Excel spreadsheets to CSV in order to 
> merge and ingest them in a newer repository.  We're experiencing an issue 
> where the processor produces a 0-byte file for some worksheets, but not 
> others.  The processor itself doesn't log any errors or post any bulletins 
> about potential data issues.
> Depending on the worksheet, this is happening with 1/3 to 1/2 of the files 
> processed, so we're losing quite a bit of that data.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-8534) ConvertExcelToCSVProcessor outputting 0-byte files

[John Wise (Jira)]

John Wise created NIFI-8534:
---

 Summary: ConvertExcelToCSVProcessor outputting 0-byte files
 Key: NIFI-8534
 URL: https://issues.apache.org/jira/browse/NIFI-8534
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Reporter: John Wise


We're converting a cache of legacy Excel spreadsheets to CSV in order to merge 
and ingest them in a newer repository.  We're experiencing an issue where the 
processor produces a 0-byte file for some worksheets, but not others.  The 
processor itself doesn't log any errors or post any bulletins about potential 
data issues.

Depending on the worksheet, this is happening with 1/3 to 1/2 of the files 
processed, so we're losing quite a bit of that data.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (NIFI-8333) Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long running transactions) and misleading error-message

[Andreas Klingenstein (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343263#comment-17343263
 ] 

Andreas Klingenstein edited comment on NIFI-8333 at 5/12/21, 2:03 PM:
--

I was able to reproduce the issue in 1.13.2. Same steps as mentioned above.

About your comment _"The stack traces in the attachment indicate that the 
truststore configured for use with InvokeHTTP does not include a certificate 
authority for the HTTPS server being accessed"_. 
 We get that error only +after+ we stopped the invokeHTTP processor with 
"Terminate Task". Before that it's working fine and we get results from the 
webservice for many thousand requests on all our serves. There seems to happen 
something when the invokeHTTP is terminated - but only on the server(s) Nifi 
must really terminate it.

 


was (Author: aklingens):
I was able to reproduce the issue in 1.13.2. Same steps as mentioned above.

About your comment _"The stack traces in the attachment indicate that the 
truststore configured for use with InvokeHTTP does not include a certificate 
authority for the HTTPS server being accessed"_. 
 We get that error only +after+ we stopped the invokeHTTP processor with 
"Terminate Task". Before that it's working fine and we get results from the 
webservice for many thousand requests on all our serves. There seems to happen 
something when the invokeHTTP is terminated - but only on the serves(s) Nifi 
must really terminate it.

 

> Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long 
> running transactions) and misleading error-message
> --
>
> Key: NIFI-8333
> URL: https://issues.apache.org/jira/browse/NIFI-8333
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.12.1, 1.13.2
>Reporter: Andreas Klingenstein
>Priority: Major
>  Labels: https, security
> Attachments: 1_config_invokeHttp_settings.png, 
> 2_config_invokeHttp_scheduling.png, 3_config_invokeHttp_config_1.png, 
> 4_config_invokeHttp_config_2.png, 5_config_invokeHttp_config_3.png, 
> 6_sslcontextservice_settings.png, 7_sslcontextservice_properties.png, 
> 8_config_invokeHttp_Terminate_Task.png, SSLHandshakeException.txt
>
>
> There seems to be an issue with InvokeHTTP 1.12.1 or 
> StandardRestrictedSSLContextService 1.12.1 or both.
> We observed a lot of "SSLHandshakeException"s (...) "unable to find valid 
> certification path to requested target" in some situations.
> At first we had a very close look into our keystore which is used in our 
> StandardRestrictedSSLContextService and made sure everything was fine. In the 
> end we figured out we had used an older, no longer valid oAuth token and the 
> webservice simlpy had rejected our request.
> Then we got the "SSLHandshakeException"s out of the blue in situations of 
> very intense testing where we started and stopped tests runs over and over 
> again. The oAuth token was still valid and other processors of the same kind 
> and even exact copies of the now failing processor worked fine
> We discovered the following steps to reproduce the issue in our environment 
> (Nifi 1.12.1 & 1.13.2):
>  - create input data for the InvokeHTTP
>  - make sure the contacted endpoint does not respond but have a high timeout 
> setting in InvokeHttp to be able to terminate the processor
>  - start InvokeHTTP
>  - let it run for some time
>  - stop the InvokeHTTP-processor
>  - "Terminate" should be available in the context menu if there are still 
> open requests waiting for an answer (or timeout)
>  - terminate it
>  - wait until all tasks are finally stopped
>  - start the InvokeHTTP again
> "SSLHandshakeException" Errors should be visible now in the nifi-app.log on 
> all machines where the invokehttp-processor had to terminate some tasks. 
> They'll
>  stay until you restart those nifi instances
> Our configuration
>  - Cluster: 3 server, one nifi instance each
>  - nifi in "http-mode" on port 8081 (nifi-properties)
> more details in the screenshots
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (NIFI-8333) Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long running transactions) and misleading error-message

[Andreas Klingenstein (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343263#comment-17343263
 ] 

Andreas Klingenstein edited comment on NIFI-8333 at 5/12/21, 2:00 PM:
--

I was able to reproduce the issue in 1.13.2. Same steps as mentioned above.

About your comment _"The stack traces in the attachment indicate that the 
truststore configured for use with InvokeHTTP does not include a certificate 
authority for the HTTPS server being accessed"_. 
 We get that error only +after+ we stopped the invokeHTTP processor with 
"Terminate Task". Before that it's working fine and we get results from the 
webservice for many thousand requests on all our serves. There seems to happen 
something when the invokeHTTP is terminated - but only on the serves(s) Nifi 
must really terminate it.

 


was (Author: aklingens):
I was able to reproduce the issue in 1.13.2. Same steps as mentioned above.

About your comment _"The stack traces in the attachment indicate that the 
truststore configured for use with InvokeHTTP does not include a certificate 
authority for the HTTPS server being accessed"_. 
We get that error only +after+ we canceled the invokeHTTP processor with 
"Terminate Task". Before that it's working fine and we get results from the 
webservice for many thousand requests on all our serves. There seems to happen 
something when the invokeHTTP is terminated - but only on the serves(s) Nifi 
must really terminate it.

 

> Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long 
> running transactions) and misleading error-message
> --
>
> Key: NIFI-8333
> URL: https://issues.apache.org/jira/browse/NIFI-8333
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.12.1, 1.13.2
>Reporter: Andreas Klingenstein
>Priority: Major
>  Labels: https, security
> Attachments: 1_config_invokeHttp_settings.png, 
> 2_config_invokeHttp_scheduling.png, 3_config_invokeHttp_config_1.png, 
> 4_config_invokeHttp_config_2.png, 5_config_invokeHttp_config_3.png, 
> 6_sslcontextservice_settings.png, 7_sslcontextservice_properties.png, 
> 8_config_invokeHttp_Terminate_Task.png, SSLHandshakeException.txt
>
>
> There seems to be an issue with InvokeHTTP 1.12.1 or 
> StandardRestrictedSSLContextService 1.12.1 or both.
> We observed a lot of "SSLHandshakeException"s (...) "unable to find valid 
> certification path to requested target" in some situations.
> At first we had a very close look into our keystore which is used in our 
> StandardRestrictedSSLContextService and made sure everything was fine. In the 
> end we figured out we had used an older, no longer valid oAuth token and the 
> webservice simlpy had rejected our request.
> Then we got the "SSLHandshakeException"s out of the blue in situations of 
> very intense testing where we started and stopped tests runs over and over 
> again. The oAuth token was still valid and other processors of the same kind 
> and even exact copies of the now failing processor worked fine
> We discovered the following steps to reproduce the issue in our environment 
> (Nifi 1.12.1 & 1.13.2):
>  - create input data for the InvokeHTTP
>  - make sure the contacted endpoint does not respond but have a high timeout 
> setting in InvokeHttp to be able to terminate the processor
>  - start InvokeHTTP
>  - let it run for some time
>  - stop the InvokeHTTP-processor
>  - "Terminate" should be available in the context menu if there are still 
> open requests waiting for an answer (or timeout)
>  - terminate it
>  - wait until all tasks are finally stopped
>  - start the InvokeHTTP again
> "SSLHandshakeException" Errors should be visible now in the nifi-app.log on 
> all machines where the invokehttp-processor had to terminate some tasks. 
> They'll
>  stay until you restart those nifi instances
> Our configuration
>  - Cluster: 3 server, one nifi instance each
>  - nifi in "http-mode" on port 8081 (nifi-properties)
> more details in the screenshots
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-8333) Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long running transactions) and misleading error-message

[Andreas Klingenstein (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343263#comment-17343263
 ] 

Andreas Klingenstein commented on NIFI-8333:


I was able to reproduce the issue in 1.13.2. Same steps as mentioned above.

About your comment _"The stack traces in the attachment indicate that the 
truststore configured for use with InvokeHTTP does not include a certificate 
authority for the HTTPS server being accessed"_. 
We get that error only +after+ we canceled the invokeHTTP processor with 
"Terminate Task". Before that it's working fine and we get results from the 
webservice for many thousand requests on all our serves. There seems to happen 
something when the invokeHTTP is terminated - but only on the serves(s) Nifi 
must really terminate it.

 

> Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long 
> running transactions) and misleading error-message
> --
>
> Key: NIFI-8333
> URL: https://issues.apache.org/jira/browse/NIFI-8333
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.12.1, 1.13.2
>Reporter: Andreas Klingenstein
>Priority: Major
>  Labels: https, security
> Attachments: 1_config_invokeHttp_settings.png, 
> 2_config_invokeHttp_scheduling.png, 3_config_invokeHttp_config_1.png, 
> 4_config_invokeHttp_config_2.png, 5_config_invokeHttp_config_3.png, 
> 6_sslcontextservice_settings.png, 7_sslcontextservice_properties.png, 
> 8_config_invokeHttp_Terminate_Task.png, SSLHandshakeException.txt
>
>
> There seems to be an issue with InvokeHTTP 1.12.1 or 
> StandardRestrictedSSLContextService 1.12.1 or both.
> We observed a lot of "SSLHandshakeException"s (...) "unable to find valid 
> certification path to requested target" in some situations.
> At first we had a very close look into our keystore which is used in our 
> StandardRestrictedSSLContextService and made sure everything was fine. In the 
> end we figured out we had used an older, no longer valid oAuth token and the 
> webservice simlpy had rejected our request.
> Then we got the "SSLHandshakeException"s out of the blue in situations of 
> very intense testing where we started and stopped tests runs over and over 
> again. The oAuth token was still valid and other processors of the same kind 
> and even exact copies of the now failing processor worked fine
> We discovered the following steps to reproduce the issue in our environment 
> (Nifi 1.12.1 & 1.13.2):
>  - create input data for the InvokeHTTP
>  - make sure the contacted endpoint does not respond but have a high timeout 
> setting in InvokeHttp to be able to terminate the processor
>  - start InvokeHTTP
>  - let it run for some time
>  - stop the InvokeHTTP-processor
>  - "Terminate" should be available in the context menu if there are still 
> open requests waiting for an answer (or timeout)
>  - terminate it
>  - wait until all tasks are finally stopped
>  - start the InvokeHTTP again
> "SSLHandshakeException" Errors should be visible now in the nifi-app.log on 
> all machines where the invokehttp-processor had to terminate some tasks. 
> They'll
>  stay until you restart those nifi instances
> Our configuration
>  - Cluster: 3 server, one nifi instance each
>  - nifi in "http-mode" on port 8081 (nifi-properties)
> more details in the screenshots
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-8333) Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long running transactions) and misleading error-message

[Andreas Klingenstein (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andreas Klingenstein updated NIFI-8333:
---
Description: 
There seems to be an issue with InvokeHTTP 1.12.1 or 
StandardRestrictedSSLContextService 1.12.1 or both.

We observed a lot of "SSLHandshakeException"s (...) "unable to find valid 
certification path to requested target" in some situations.

At first we had a very close look into our keystore which is used in our 
StandardRestrictedSSLContextService and made sure everything was fine. In the 
end we figured out we had used an older, no longer valid oAuth token and the 
webservice simlpy had rejected our request.

Then we got the "SSLHandshakeException"s out of the blue in situations of very 
intense testing where we started and stopped tests runs over and over again. 
The oAuth token was still valid and other processors of the same kind and even 
exact copies of the now failing processor worked fine

We discovered the following steps to reproduce the issue in our environment 
(Nifi 1.12.1 & 1.13.2):
 - create input data for the InvokeHTTP
 - make sure the contacted endpoint does not respond but have a high timeout 
setting in InvokeHttp to be able to terminate the processor
 - start InvokeHTTP
 - let it run for some time
 - stop the InvokeHTTP-processor
 - "Terminate" should be available in the context menu if there are still open 
requests waiting for an answer (or timeout)
 - terminate it
 - wait until all tasks are finally stopped
 - start the InvokeHTTP again

"SSLHandshakeException" Errors should be visible now in the nifi-app.log on all 
machines where the invokehttp-processor had to terminate some tasks. They'll
 stay until you restart those nifi instances

Our configuration
 - Cluster: 3 server, one nifi instance each
 - nifi in "http-mode" on port 8081 (nifi-properties)

more details in the screenshots

 

  was:
There seems to be an issue with InvokeHTTP 1.12.1 or 
StandardRestrictedSSLContextService 1.12.1 or both.

We observed a lot of "SSLHandshakeException"s (...) "unable to find valid 
certification path to requested target" in some situations.


 At first we had a very close look into our keystore which is used in our 
StandardRestrictedSSLContextService and made sure everything was fine. In the 
end we figured out we had used an older, no longer valid oAuth token and the 
webservice simlpy had rejected our request.

Then we got the "SSLHandshakeException"s out of the blue in situations of very 
intense testing where we started and stopped tests runs over and over again. 
The oAuth token was still valid and other processors of the same kind and even 
exact copies of the now failing processor worked fine

We discovered the following steps to reproduce the issue in our environment 
(Nifi 1.12.1):

- create input data for the InvokeHTTP
 - make sure the contacted endpoint does not respond but have a high timeout 
setting in InvokeHttp to be able to terminate the processor
 - start InvokeHTTP 
 - let it run for some time
 - stop the InvokeHTTP-processor 
 - "Terminate" should be available in the context menu if there are still open 
requests waiting for an answer (or timeout)
 - terminate it
 - wait until all tasks are finally stopped
 - start the InvokeHTTP again

"SSLHandshakeException" Errors should be visible now in the nifi-app.log on all 
machines where the invokehttp-processor had to terminate some tasks. They'll
 stay until you restart those nifi instances

Our configuration 
 - Cluster: 3 server, one nifi instance each
 - nifi in "http-mode" on port 8081 (nifi-properties)

more details in the screenshots

 


> Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long 
> running transactions) and misleading error-message
> --
>
> Key: NIFI-8333
> URL: https://issues.apache.org/jira/browse/NIFI-8333
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.12.1, 1.13.2
>Reporter: Andreas Klingenstein
>Priority: Major
>  Labels: https, security
> Attachments: 1_config_invokeHttp_settings.png, 
> 2_config_invokeHttp_scheduling.png, 3_config_invokeHttp_config_1.png, 
> 4_config_invokeHttp_config_2.png, 5_config_invokeHttp_config_3.png, 
> 6_sslcontextservice_settings.png, 7_sslcontextservice_properties.png, 
> 8_config_invokeHttp_Terminate_Task.png, SSLHandshakeException.txt
>
>
> There seems to be an issue with InvokeHTTP 1.12.1 or 
> StandardRestrictedSSLContextService 1.12.1 or both.
> We observed a lot of "SSLHandshakeException"s (...) "unable to find valid 
> certification path to requested target" in some situations.
> At first we had a very close look into our keystore which is used in our 
> StandardRestrictedSSLConte

[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1066: MINIFICPP-1532: Create a processor to capture resource consumption me…

[GitBox]

martinzink commented on a change in pull request #1066:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1066#discussion_r631058529



##
File path: libminifi/test/unit/MemoryUsageTest.cpp
##
@@ -20,22 +20,22 @@
 #include "utils/OsUtils.h"
 #include "../TestBase.h"
 
-TEST_CASE("Test memory usage", "[testmemoryusage]") {
+TEST_CASE("Test Physical memory usage", "[testphysicalmemoryusage]") {
   constexpr bool cout_enabled = true;

Review comment:
   These hardware/platform specific functions are hard to test (because 
they depend on a lot of things).
   I added cout the these tests so one can manually check them (with 3rd party 
tools) that they return the correct data.
   I also wanted to give a simple switch, if sometime we want to mute these 
tests.

##
File path: extensions/pdh/PerformanceDataMonitor.cpp
##
@@ -0,0 +1,283 @@
+/**
+ * @file GenerateFlowFile.cpp
+ * GenerateFlowFile class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "PerformanceDataMonitor.h"
+#include "PDHCounters.h"
+#include "MemoryConsumptionCounter.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Relationship PerformanceDataMonitor::Success("success", "All files are 
routed to success");
+
+core::Property PerformanceDataMonitor::PredefinedGroups(
+core::PropertyBuilder::createProperty("Predefined Groups")->
+withDescription("Comma separated list to which predefined groups should be 
included")->
+withDefaultValue("")->build());
+
+core::Property PerformanceDataMonitor::CustomPDHCounters(
+core::PropertyBuilder::createProperty("Custom PDH Counters")->
+withDescription("Comma separated list of PDHCounters to collect from")->
+withDefaultValue("")->build());
+
+core::Property PerformanceDataMonitor::OutputFormatProperty(
+core::PropertyBuilder::createProperty("Output format")->
+withDescription("Format of the created flowfiles")->
+withAllowableValue(JSON_FORMAT_STR)->
+withAllowableValue(OPEN_TELEMETRY_FORMAT_STR)->
+withDefaultValue(JSON_FORMAT_STR)->build());
+
+PerformanceDataMonitor::~PerformanceDataMonitor() {
+  if (pdh_query_ != nullptr)
+PdhCloseQuery(pdh_query_);
+  for (PerformanceDataCounter* resource_consumption_counter : 
resource_consumption_counters_) {
+delete resource_consumption_counter;
+  }
+}
+
+void PerformanceDataMonitor::onSchedule(const 
std::shared_ptr& context, const 
std::shared_ptr& sessionFactory) {
+  setupMembersFromProperties(context);
+
+  std::vector valid_counters;
+
+  for (PerformanceDataCounter* counter : resource_consumption_counters_) {
+PDHCounterBase* pdh_counter = dynamic_cast (counter);
+if (pdh_counter != nullptr) {
+  if (pdh_query_ == nullptr)
+PdhOpenQuery(NULL, NULL, &pdh_query_);
+  PDH_STATUS add_to_query_result = pdh_counter->addToQuery(pdh_query_);
+  if (add_to_query_result != ERROR_SUCCESS) {
+logger_->log_error(("Error adding " + pdh_counter->getName() + " to 
query: " + std::to_string(add_to_query_result)).c_str());
+delete counter;
+  } else {
+valid_counters.push_back(counter);
+  }
+} else {
+  valid_counters.push_back(counter);
+}
+  }
+  resource_consumption_counters_ = valid_counters;
+  PdhCollectQueryData(pdh_query_);
+}
+
+void PerformanceDataMonitor::onTrigger(core::ProcessContext* context, 
core::ProcessSession* session) {
+  if (resource_consumption_counters_.empty()) {
+logger_->log_error("No valid counters for PerformanceDataMonitor");
+return;
+  }
+
+  std::shared_ptr flowFile = session->create();
+  if (!flowFile) {
+logger_->log_error("Failed to create flowfile!");
+return;
+  }
+
+  if (pdh_query_ != nullptr)
+PdhCollectQueryData(pdh_query_);
+
+  rapidjson::Document root = rapidjson::Document(rapidjson::kObjectType);
+  rapidjson::Value& body = prepareJSONBody(root);
+  for (PerformanceDataCounter* counter : resource_consumption_counters_) {
+PDHCounterBase* pdh_counter = dynamic_cast(counter);
+if (pdh_counter != nullptr && pdh_counter->collectData() != ERROR_SUCCESS)
+  continue;
+counter->addToJson(body, root.G

[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1066: MINIFICPP-1532: Create a processor to capture resource consumption me…

[GitBox]

martinzink commented on a change in pull request #1066:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1066#discussion_r631058212



##
File path: extensions/pdh/tests/PerformanceDataMonitorTests.cpp
##
@@ -0,0 +1,268 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "TestBase.h"
+#include "processors/PutFile.h"
+#include "utils/file/FileUtils.h"
+#include "PerformanceDataMonitor.h"
+#include "rapidjson/filereadstream.h"
+
+using PutFile = org::apache::nifi::minifi::processors::PutFile;
+using PerformanceDataMonitor = 
org::apache::nifi::minifi::processors::PerformanceDataMonitor;
+using PerformanceDataCounter = 
org::apache::nifi::minifi::processors::PerformanceDataCounter;
+
+class PerformanceDataMonitorTester {
+ public:
+  PerformanceDataMonitorTester() {
+LogTestController::getInstance().setTrace();
+dir_ = test_controller_.createTempDirectory("/tmp/gt.XX");
+plan_ = test_controller_.createPlan();
+performance_monitor_ = plan_->addProcessor("PerformanceDataMonitor", 
"pdhsys");
+putfile_ = plan_->addProcessor("PutFile", "putfile", 
core::Relationship("success", "description"), true);
+plan_->setProperty(putfile_, PutFile::Directory.getName(), dir_);
+  }
+
+  void runProcessors() {
+plan_->runNextProcessor();  // PerformanceMonitor
+std::this_thread::sleep_for(std::chrono::milliseconds(200));
+plan_->runCurrentProcessor();   // PerformanceMonitor
+plan_->runNextProcessor();  // PutFile
+plan_->runCurrentProcessor();   // PutFile
+  }
+
+  void setPerformanceMonitorProperty(core::Property property, const 
std::string& value) {

Review comment:
   changed it in 
[7422bb3](https://github.com/martinzink/nifi-minifi-cpp/commit/7422bb3217a1770e3a7f272436f25be5ca417377)

##
File path: extensions/pdh/tests/PerformanceDataMonitorTests.cpp
##
@@ -0,0 +1,268 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "TestBase.h"
+#include "processors/PutFile.h"
+#include "utils/file/FileUtils.h"
+#include "PerformanceDataMonitor.h"
+#include "rapidjson/filereadstream.h"
+
+using PutFile = org::apache::nifi::minifi::processors::PutFile;
+using PerformanceDataMonitor = 
org::apache::nifi::minifi::processors::PerformanceDataMonitor;
+using PerformanceDataCounter = 
org::apache::nifi::minifi::processors::PerformanceDataCounter;
+
+class PerformanceDataMonitorTester {
+ public:
+  PerformanceDataMonitorTester() {
+LogTestController::getInstance().setTrace();
+dir_ = test_controller_.createTempDirectory("/tmp/gt.XX");

Review comment:
   changed it in 
[7422bb3](https://github.com/martinzink/nifi-minifi-cpp/commit/7422bb3217a1770e3a7f272436f25be5ca417377)

##
File path: extensions/pdh/tests/PerformanceDataCounterTests.cpp
##
@@ -0,0 +1,185 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required 

[jira] [Updated] (NIFI-8333) Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long running transactions) and misleading error-message

[Andreas Klingenstein (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andreas Klingenstein updated NIFI-8333:
---
Affects Version/s: 1.13.2

> Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long 
> running transactions) and misleading error-message
> --
>
> Key: NIFI-8333
> URL: https://issues.apache.org/jira/browse/NIFI-8333
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.12.1, 1.13.2
>Reporter: Andreas Klingenstein
>Priority: Major
>  Labels: https, security
> Attachments: 1_config_invokeHttp_settings.png, 
> 2_config_invokeHttp_scheduling.png, 3_config_invokeHttp_config_1.png, 
> 4_config_invokeHttp_config_2.png, 5_config_invokeHttp_config_3.png, 
> 6_sslcontextservice_settings.png, 7_sslcontextservice_properties.png, 
> 8_config_invokeHttp_Terminate_Task.png, SSLHandshakeException.txt
>
>
> There seems to be an issue with InvokeHTTP 1.12.1 or 
> StandardRestrictedSSLContextService 1.12.1 or both.
> We observed a lot of "SSLHandshakeException"s (...) "unable to find valid 
> certification path to requested target" in some situations.
>  At first we had a very close look into our keystore which is used in our 
> StandardRestrictedSSLContextService and made sure everything was fine. In the 
> end we figured out we had used an older, no longer valid oAuth token and the 
> webservice simlpy had rejected our request.
> Then we got the "SSLHandshakeException"s out of the blue in situations of 
> very intense testing where we started and stopped tests runs over and over 
> again. The oAuth token was still valid and other processors of the same kind 
> and even exact copies of the now failing processor worked fine
> We discovered the following steps to reproduce the issue in our environment 
> (Nifi 1.12.1):
> - create input data for the InvokeHTTP
>  - make sure the contacted endpoint does not respond but have a high timeout 
> setting in InvokeHttp to be able to terminate the processor
>  - start InvokeHTTP 
>  - let it run for some time
>  - stop the InvokeHTTP-processor 
>  - "Terminate" should be available in the context menu if there are still 
> open requests waiting for an answer (or timeout)
>  - terminate it
>  - wait until all tasks are finally stopped
>  - start the InvokeHTTP again
> "SSLHandshakeException" Errors should be visible now in the nifi-app.log on 
> all machines where the invokehttp-processor had to terminate some tasks. 
> They'll
>  stay until you restart those nifi instances
> Our configuration 
>  - Cluster: 3 server, one nifi instance each
>  - nifi in "http-mode" on port 8081 (nifi-properties)
> more details in the screenshots
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] JonathanKessler commented on pull request #4780: NIFI-8126 Include Total Queued Duration in ConnectionStatus metrics

[GitBox]

JonathanKessler commented on pull request #4780:
URL: https://github.com/apache/nifi/pull/4780#issuecomment-839779931


   > Thanks for the update @JonathanKessler . I do think that you addressed the 
concerns that I had in the first review. Looked over it again. There are a few 
places where the code should probably be cleaned up a bit (commented inline). 
Small changes to just make it easier to understand.
   > 
   > There's also the potential for a NullPointerException that must be 
addressed before we can merge this.
   > 
   > However, the stated goal of NIFI-8126 is to include a new metric about 
average queue duration, etc. in the stats history. This does not accomplish 
that. It updates the ConnectionStatus, but it doesn't give us anything in the 
UI to select the Average Queue Duration. The ConnectionStatusDescriptor needs 
to be updated to expose that information.
   
   I will make the requested changes. The ticket actually says to expose a 
total or an average. I went with the total, leaving it up to consuming systems 
to calculate the average. This matches other metrics available within 
ConnectionStatus such as inputBytes, queuedBytes, etc. I will look at adding 
that to the ConnectionStatusDescriptor.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] ottobackwards commented on pull request #5059: NIFI-8519 Adding HDFS support for NAR autoload

[GitBox]

ottobackwards commented on pull request #5059:
URL: https://github.com/apache/nifi/pull/5059#issuecomment-839773885


   One reason I implemented this by swapping out FileUtils to use VFS is that 
you don't need to change any of this, you just set the nifi properties paths 
that VFS urls and the rest just works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] harjitdotsingh commented on pull request #5046: Fix for #Nifi-8646 Added Support for ParQuet, ORC and DataStoreBackup.

[GitBox]

harjitdotsingh commented on pull request #5046:
URL: https://github.com/apache/nifi/pull/5046#issuecomment-839768161


   I knew that was coming. For some reasons my IDE formatted the code. Let me
   fix that. I will add that. I didn't see tests for any other cases other
   than JSON so left it but let me see if I can find some sample files for
   Parquet,ORC and DataStoreBackup.
   
   Thanks
   
   - harjit
   
   On Wed, May 12, 2021 at 7:02 AM Pierre Villard ***@***.***>
   wrote:
   
   > Hi @harjitdotsingh  - happy to review
   > these changes but can you make sure the PR is only changing what needs to
   > be changed? there is a lot of formatting related changes that are making
   > the review harder. Can we also have tests for the changes?
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > , or
   > unsubscribe
   > 

   > .
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1064: MINIFICPP-987 Improve error handling in GetFile

[GitBox]

adamdebreceni commented on a change in pull request #1064:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1064#discussion_r631034571



##
File path: libminifi/src/utils/FileReaderCallback.cpp
##
@@ -0,0 +1,72 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "utils/FileReaderCallback.h"
+
+#include 
+
+#include "Exception.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace {
+
+constexpr std::size_t BUFFER_SIZE = 4096;
+
+}  // namespace
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+
+FileReaderCallback::FileReaderCallback(const std::string& file_name)
+: logger_(core::logging::LoggerFactory::getLogger()) {
+  logger_->log_debug("Opening %s", file_name);
+  input_stream_.open(file_name.c_str(), std::fstream::in | 
std::fstream::binary);
+  if (!input_stream_.is_open()) {
+throw Exception(FILE_OPERATION_EXCEPTION, "Could not open file: " + 
file_name);
+  }
+}
+
+int64_t FileReaderCallback::process(const std::shared_ptr& 
output_stream) {
+  std::array buffer;
+
+  uint64_t num_bytes_written = 0;
+
+  while (input_stream_.good()) {
+input_stream_.read(buffer.data(), buffer.size());
+
+const auto num_bytes_read = input_stream_.gcount();
+logger_->log_trace("Read %jd bytes of input", 
std::intmax_t{num_bytes_read});
+
+const int len = gsl::narrow(num_bytes_read);
+
+output_stream->write(reinterpret_cast(buffer.data()), len);
+num_bytes_written += len;
+  }
+  input_stream_.close();
+
+  logger_->log_debug("Finished reading %" PRIu64 " bytes from the file", 
num_bytes_written);
+  return num_bytes_written;

Review comment:
   do we set the stream somewhere to throw on `failure`? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1064: MINIFICPP-987 Improve error handling in GetFile

[GitBox]

adamdebreceni commented on a change in pull request #1064:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1064#discussion_r631033715



##
File path: extensions/standard-processors/processors/GetFile.cpp
##
@@ -158,76 +147,89 @@ void GetFile::onSchedule(core::ProcessContext *context, 
core::ProcessSessionFact
   request_.inputDirectory = value;
 }
 
-void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession *session) {
-  // Perform directory list
-
+void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession* session) {
   metrics_->iterations_++;
 
-  const bool isDirEmptyBeforePoll = isListingEmpty();
-  logger_->log_debug("Is listing empty before polling directory %i", 
isDirEmptyBeforePoll);
-  if (isDirEmptyBeforePoll) {
+  const bool is_dir_empty_before_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s before polling directory", 
is_dir_empty_before_poll ? "empty" : "not empty");
+  if (is_dir_empty_before_poll) {
 if (request_.pollInterval == 0 || (utils::timeutils::getTimeMillis() - 
last_listing_time_) > request_.pollInterval) {
   performListing(request_);
   last_listing_time_.store(utils::timeutils::getTimeMillis());
 }
   }
 
-  const bool isDirEmptyAfterPoll = isListingEmpty();
-  logger_->log_debug("Is listing empty after polling directory %i", 
isDirEmptyAfterPoll);
-
-  if (!isDirEmptyAfterPoll) {
-try {
-  std::queue list;
-  pollListing(list, request_);
-  while (!list.empty()) {
-std::string fileName = list.front();
-list.pop();
-logger_->log_info("GetFile process %s", fileName);
-auto flowFile = session->create();
-if (flowFile == nullptr)
-  return;
-std::size_t found = fileName.find_last_of("/\\");
-std::string path = fileName.substr(0, found);
-std::string name = fileName.substr(found + 1);
-flowFile->setAttribute(core::SpecialFlowAttribute::FILENAME, name);
-flowFile->setAttribute(core::SpecialFlowAttribute::PATH, path);
-flowFile->addAttribute(core::SpecialFlowAttribute::ABSOLUTE_PATH, 
fileName);
-session->import(fileName, flowFile, request_.keepSourceFile);
-session->transfer(flowFile, Success);
+  const bool is_dir_empty_after_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s after polling directory", 
is_dir_empty_after_poll ? "empty" : "not empty");
+  if (is_dir_empty_after_poll) {
+yield();
+return;
+  }
+
+  std::queue list_of_file_names = pollListing(request_.batchSize);
+  while (!list_of_file_names.empty()) {
+std::string file_name = list_of_file_names.front();
+list_of_file_names.pop();
+getSingleFile(*session, file_name);
+  }
+}
+
+void GetFile::getSingleFile(core::ProcessSession& session, const std::string& 
file_name) const {
+  logger_->log_info("GetFile process %s", file_name);
+  auto flow_file = session.create();
+  gsl_Expects(flow_file);
+  std::string path;
+  std::string name;
+  std::tie(path, name) = utils::file::split_path(file_name);
+  flow_file->setAttribute(core::SpecialFlowAttribute::FILENAME, name);
+  flow_file->setAttribute(core::SpecialFlowAttribute::PATH, path);
+  flow_file->addAttribute(core::SpecialFlowAttribute::ABSOLUTE_PATH, 
file_name);
+
+  try {
+utils::FileReaderCallback file_reader_callback{file_name};
+session.write(flow_file, &file_reader_callback);
+session.transfer(flow_file, Success);
+if (!request_.keepSourceFile) {
+  auto remove_status = remove(file_name.c_str());
+  if (remove_status != 0) {
+logger_->log_error("GetFile could not delete file '%s', error %d: %s", 
file_name, errno, strerror(errno));
   }
-} catch (std::exception &exception) {
-  logger_->log_debug("GetFile Caught Exception %s", exception.what());
-  throw;
-} catch (...) {
-  throw;
 }
+  } catch (const std::exception& exception) {

Review comment:
   we could limit the kinds of exceptions we consider "recoverable" (i.e. 
discard that specific flowfile, but continue processor execution) we could go 
for `std::ios_base::failure`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] simonbence commented on pull request #5059: NIFI-8519 Adding HDFS support for NAR autoload

[GitBox]

simonbence commented on pull request #5059:
URL: https://github.com/apache/nifi/pull/5059#issuecomment-839697425


   Thanks @markap14 ! I am on incorporating your ideas


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (NIFI-8457) When Load-balanced connections are used, offloading a node may never complete

[Pierre Villard (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-8457.
--
Resolution: Fixed

> When Load-balanced connections are used, offloading a node may never complete
> -
>
> Key: NIFI-8457
> URL: https://issues.apache.org/jira/browse/NIFI-8457
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.14.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> If NiFi has a load-balanced connections, and the node is offloaded, the 
> offloaded node may sometimes always show itself as having data in a queue, 
> resulting in the offload never completing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-8457) When Load-balanced connections are used, offloading a node may never complete

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343176#comment-17343176
 ] 

ASF subversion and git services commented on NIFI-8457:
---

Commit ab8b7444b5587adfc7548a1da99c57040dc418c0 in nifi's branch 
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ab8b744 ]

NIFI-8457: Fixed bug in load balanced connections that can result in the node 
never completing OFFLOAD action. Also fixed issue in which data destined for a 
disconnected/offloaded node was never rebalanced even for partitioning 
strategies that call for rebalancing on failure

Signed-off-by: Pierre Villard 

This closes #5019.


> When Load-balanced connections are used, offloading a node may never complete
> -
>
> Key: NIFI-8457
> URL: https://issues.apache.org/jira/browse/NIFI-8457
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.14.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> If NiFi has a load-balanced connections, and the node is offloaded, the 
> offloaded node may sometimes always show itself as having data in a queue, 
> resulting in the offload never completing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] asfgit closed pull request #5019: NIFI-8457: Fixed bug in load balanced connections that can result in …

[GitBox]

asfgit closed pull request #5019:
URL: https://github.com/apache/nifi/pull/5019


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] pvillard31 commented on pull request #5033: Expanded property description

[GitBox]

pvillard31 commented on pull request #5033:
URL: https://github.com/apache/nifi/pull/5033#issuecomment-839682048


   Hey @dennisjaheruddin - is there is a JIRA for this change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] pvillard31 commented on pull request #5046: Fix for #Nifi-8646 Added Support for ParQuet, ORC and DataStoreBackup.

[GitBox]

pvillard31 commented on pull request #5046:
URL: https://github.com/apache/nifi/pull/5046#issuecomment-839680947


   Hi @harjitdotsingh - happy to review these changes but can you make sure the 
PR is only changing what needs to be changed? there is a lot of formatting 
related changes that are making the review harder. Can we also have tests for 
the changes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-1465) Upgrade encryption of sensitive properties

[Pierre Villard (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-1465:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Upgrade encryption of sensitive properties
> --
>
> Key: NIFI-1465
> URL: https://issues.apache.org/jira/browse/NIFI-1465
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.5.0
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Major
>  Labels: encryption, security
> Fix For: 1.14.0
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> Currently, NiFi accepts a password and encryption algorithm in 
> {{nifi.properties}} which are used to encrypt all sensitive processor 
> properties throughout the application. The password defaults to empty and the 
> algorithm defaults to {{PBEWITHMD5AND256BITAES-CBC-OPENSSL}}. This algorithm:
> * uses a digest function ({{MD5}}) which is not cryptographically secure 
> [1][2][3][4]
> * uses a single iteration count [5][6]
> * limits password input to 16 characters on JVMs without the unlimited 
> strength cryptographic jurisdiction policy files installed [NIFI-1255]
> all of which combine to make it extremely insecure. We should change the 
> default algorithm to use a strong key derivation function (KDF) [7] which 
> will properly derive a key to protect the sensitive properties. 
> Because existing systems have already encrypted the properties using a key 
> derived from the original settings, we should provide a translation/upgrade 
> utility to seamlessly convert the stored values from the old password & 
> algorithm combination to the new. 
> [1] http://security.stackexchange.com/a/19908/16485
> [2] http://security.stackexchange.com/a/31846/16485
> [3] 
> http://security.stackexchange.com/questions/52461/how-weak-is-md5-as-a-password-hashing-function
> [4] http://security.stackexchange.com/a/31410/16485
> [5] http://security.stackexchange.com/a/29139/16485
> [6] https://www.openssl.org/docs/manmaster/crypto/EVP_BytesToKey.html
> [7] 
> https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-8246) Set Default Sensitive Properties Algorithm with Improved KDF and Encryption

[Pierre Villard (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-8246:
-
Fix Version/s: 1.14.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Set Default Sensitive Properties Algorithm with Improved KDF and Encryption
> ---
>
> Key: NIFI-8246
> URL: https://issues.apache.org/jira/browse/NIFI-8246
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Security
>Affects Versions: 1.13.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.14.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The default Sensitive Properties Algorithm specified using 
> {{nifi.sensitive.properties.algorithm}} in {{nifi.properties}} has been 
> {{PBEWITHMD5AND256BITAES-CBC-OPENSSL}} since early release versions.  This 
> default value relies on the {{NiFiLegacyCipherProvider}}, which is 
> deprecated.  The {{NiFiLegacyCipherProvider}} uses the MD5 hash algorithm 
> with 1000 iterations and a random salt.  This algorithm configuration also 
> specifies AES with CBC, which does not provide Authenticated Encryption with 
> Associated Data.
> Recent NiFi versions support the Argon2 secure hashing algorithm and AES in 
> Galois/Counter Mode.  NIFI-7668 introduces support for additional secure 
> hashing algorithms along with support for AES-GCM.  One of the options that 
> incorporates an improved Key Derivation Function and AES-GCM should be set as 
> the default sensitive properties algorithm in order to provide greater 
> security for encryption of sensitive properties.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-8246) Set Default Sensitive Properties Algorithm with Improved KDF and Encryption

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343171#comment-17343171
 ] 

ASF subversion and git services commented on NIFI-8246:
---

Commit 09e54c1dad09a6b91d5e9a08ee53de42afa56c53 in nifi's branch 
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=09e54c1 ]

NIFI-8246 Set NIFI_PBKDF2_AES_GCM_256 as default properties encryption algorithm

Signed-off-by: Pierre Villard 

This closes #5055.


> Set Default Sensitive Properties Algorithm with Improved KDF and Encryption
> ---
>
> Key: NIFI-8246
> URL: https://issues.apache.org/jira/browse/NIFI-8246
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Security
>Affects Versions: 1.13.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The default Sensitive Properties Algorithm specified using 
> {{nifi.sensitive.properties.algorithm}} in {{nifi.properties}} has been 
> {{PBEWITHMD5AND256BITAES-CBC-OPENSSL}} since early release versions.  This 
> default value relies on the {{NiFiLegacyCipherProvider}}, which is 
> deprecated.  The {{NiFiLegacyCipherProvider}} uses the MD5 hash algorithm 
> with 1000 iterations and a random salt.  This algorithm configuration also 
> specifies AES with CBC, which does not provide Authenticated Encryption with 
> Associated Data.
> Recent NiFi versions support the Argon2 secure hashing algorithm and AES in 
> Galois/Counter Mode.  NIFI-7668 introduces support for additional secure 
> hashing algorithms along with support for AES-GCM.  One of the options that 
> incorporates an improved Key Derivation Function and AES-GCM should be set as 
> the default sensitive properties algorithm in order to provide greater 
> security for encryption of sensitive properties.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] asfgit closed pull request #5055: NIFI-8246 Set NIFI_PBKDF2_AES_GCM_256 as Default Properties Algorithm

[GitBox]

asfgit closed pull request #5055:
URL: https://github.com/apache/nifi/pull/5055


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-8502) Upgrade Spring Framework to latest 5.x version

[Pierre Villard (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-8502:
-
Fix Version/s: 1.14.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Upgrade Spring Framework to latest 5.x version
> --
>
> Key: NIFI-8502
> URL: https://issues.apache.org/jira/browse/NIFI-8502
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.13.2
>Reporter: Joseph Gresock
>Assignee: David Handermann
>Priority: Major
>  Labels: security, springframework, springsecurity
> Fix For: 1.14.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In order to take advantage of more recent Spring services like Spring Vault 
> (relies on 5.3.5), we should consider upgrading spring-core and other Spring 
> dependencies to the latest 5.x release.  Note that this would be a major 
> version upgrade for most Spring dependencies.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-8502) Upgrade Spring Framework to latest 5.x version

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-8502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343170#comment-17343170
 ] 

ASF subversion and git services commented on NIFI-8502:
---

Commit 4db50f2f40480c852342796d999f71dff316eaa2 in nifi's branch 
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4db50f2 ]

NIFI-8502 Upgraded Spring Framework to 5.3.6

- Upgraded Spring Framework references from version 4.3.30 to 5.3.6
- Upgraded Spring Security from version 4.2.20 to 5.4.6
- Upgraded Spring Data Redis from 2.1.16 to 2.5.0
- Upgraded Jedis from 2.9.0 to 3.6.0 to match Spring Data Redis 2.5.0
- Upgraded Easy Rules from 3.4.0 to 4.1.0 to support Spring 5
- Upgraded Hortonworks Schema Registry Client from 0.8.1 to 0.9.1 to support 
Spring 5
- Refactored ThreadPoolRequestReplicatorFactoryBean to implement DisposableBean 
to handle executor shutdown

Signed-off-by: Pierre Villard 

This closes #5066.


> Upgrade Spring Framework to latest 5.x version
> --
>
> Key: NIFI-8502
> URL: https://issues.apache.org/jira/browse/NIFI-8502
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.13.2
>Reporter: Joseph Gresock
>Assignee: David Handermann
>Priority: Major
>  Labels: security, springframework, springsecurity
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In order to take advantage of more recent Spring services like Spring Vault 
> (relies on 5.3.5), we should consider upgrading spring-core and other Spring 
> dependencies to the latest 5.x release.  Note that this would be a major 
> version upgrade for most Spring dependencies.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] asfgit closed pull request #5066: NIFI-8502 Upgrade Spring Framework to 5.3.6

[GitBox]

asfgit closed pull request #5066:
URL: https://github.com/apache/nifi/pull/5066


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] pvillard31 commented on pull request #5066: NIFI-8502 Upgrade Spring Framework to 5.3.6

[GitBox]

pvillard31 commented on pull request #5066:
URL: https://github.com/apache/nifi/pull/5066#issuecomment-839672737


   Built the PR and ran some basic flows, everything is looking good on my side 
AFAICT. Merging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] turcsanyip commented on a change in pull request #4822: NIFI-2892 Implement AWS Kinesis Stream Get Processor

[GitBox]

turcsanyip commented on a change in pull request #4822:
URL: https://github.com/apache/nifi/pull/4822#discussion_r630918349



##
File path: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/kinesis/stream/ConsumeKinesisStream.java
##
@@ -0,0 +1,726 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.kinesis.stream;
+
+import 
com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessorFactory;
+import 
com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
+import 
com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
+import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker;
+import 
com.amazonaws.services.kinesis.clientlibrary.lib.worker.WorkerStateChangeListener;
+import com.amazonaws.services.kinesis.metrics.impl.NullMetricsFactory;
+import org.apache.commons.beanutils.BeanUtilsBean;
+import org.apache.commons.beanutils.ConvertUtilsBean2;
+import org.apache.commons.beanutils.FluentPropertyBeanIntrospector;
+import org.apache.commons.beanutils.PropertyUtilsBean;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.DynamicProperties;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor;
+import 
org.apache.nifi.processors.aws.kinesis.stream.record.AbstractKinesisRecordProcessor;
+import 
org.apache.nifi.processors.aws.kinesis.stream.record.KinesisRecordProcessorRaw;
+import 
org.apache.nifi.processors.aws.kinesis.stream.record.KinesisRecordProcessorRecord;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.RecordFieldType;
+
+import java.lang.reflect.InvocationTargetException;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.time.LocalDateTime;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@TriggerSerially
+@Tags({"amazon", "aws", "kinesis", "consume", "stream"})
+@CapabilityDescripti

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #1064: MINIFICPP-987 Improve error handling in GetFile

[GitBox]

arpadboda commented on a change in pull request #1064:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1064#discussion_r630913672



##
File path: extensions/standard-processors/processors/GetFile.cpp
##
@@ -158,76 +147,89 @@ void GetFile::onSchedule(core::ProcessContext *context, 
core::ProcessSessionFact
   request_.inputDirectory = value;
 }
 
-void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession *session) {
-  // Perform directory list
-
+void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession* session) {
   metrics_->iterations_++;
 
-  const bool isDirEmptyBeforePoll = isListingEmpty();
-  logger_->log_debug("Is listing empty before polling directory %i", 
isDirEmptyBeforePoll);
-  if (isDirEmptyBeforePoll) {
+  const bool is_dir_empty_before_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s before polling directory", 
is_dir_empty_before_poll ? "empty" : "not empty");

Review comment:
   No, definitely not, leave it as it is. :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #1064: MINIFICPP-987 Improve error handling in GetFile

[GitBox]

arpadboda commented on a change in pull request #1064:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1064#discussion_r630913672



##
File path: extensions/standard-processors/processors/GetFile.cpp
##
@@ -158,76 +147,89 @@ void GetFile::onSchedule(core::ProcessContext *context, 
core::ProcessSessionFact
   request_.inputDirectory = value;
 }
 
-void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession *session) {
-  // Perform directory list
-
+void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession* session) {
   metrics_->iterations_++;
 
-  const bool isDirEmptyBeforePoll = isListingEmpty();
-  logger_->log_debug("Is listing empty before polling directory %i", 
isDirEmptyBeforePoll);
-  if (isDirEmptyBeforePoll) {
+  const bool is_dir_empty_before_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s before polling directory", 
is_dir_empty_before_poll ? "empty" : "not empty");

Review comment:
   No, deffinitely not, leave it as it is. :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #1064: MINIFICPP-987 Improve error handling in GetFile

[GitBox]

arpadboda commented on a change in pull request #1064:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1064#discussion_r630913410



##
File path: extensions/standard-processors/processors/GetFile.cpp
##
@@ -158,76 +147,89 @@ void GetFile::onSchedule(core::ProcessContext *context, 
core::ProcessSessionFact
   request_.inputDirectory = value;
 }
 
-void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession *session) {
-  // Perform directory list
-
+void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession* session) {
   metrics_->iterations_++;
 
-  const bool isDirEmptyBeforePoll = isListingEmpty();
-  logger_->log_debug("Is listing empty before polling directory %i", 
isDirEmptyBeforePoll);
-  if (isDirEmptyBeforePoll) {
+  const bool is_dir_empty_before_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s before polling directory", 
is_dir_empty_before_poll ? "empty" : "not empty");
+  if (is_dir_empty_before_poll) {
 if (request_.pollInterval == 0 || (utils::timeutils::getTimeMillis() - 
last_listing_time_) > request_.pollInterval) {
   performListing(request_);
   last_listing_time_.store(utils::timeutils::getTimeMillis());
 }
   }
 
-  const bool isDirEmptyAfterPoll = isListingEmpty();
-  logger_->log_debug("Is listing empty after polling directory %i", 
isDirEmptyAfterPoll);
-
-  if (!isDirEmptyAfterPoll) {
-try {
-  std::queue list;
-  pollListing(list, request_);
-  while (!list.empty()) {
-std::string fileName = list.front();
-list.pop();
-logger_->log_info("GetFile process %s", fileName);
-auto flowFile = session->create();
-if (flowFile == nullptr)
-  return;
-std::size_t found = fileName.find_last_of("/\\");
-std::string path = fileName.substr(0, found);
-std::string name = fileName.substr(found + 1);
-flowFile->setAttribute(core::SpecialFlowAttribute::FILENAME, name);
-flowFile->setAttribute(core::SpecialFlowAttribute::PATH, path);
-flowFile->addAttribute(core::SpecialFlowAttribute::ABSOLUTE_PATH, 
fileName);
-session->import(fileName, flowFile, request_.keepSourceFile);
-session->transfer(flowFile, Success);
+  const bool is_dir_empty_after_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s after polling directory", 
is_dir_empty_after_poll ? "empty" : "not empty");
+  if (is_dir_empty_after_poll) {
+yield();
+return;
+  }
+
+  std::queue list_of_file_names = pollListing(request_.batchSize);
+  while (!list_of_file_names.empty()) {
+std::string file_name = list_of_file_names.front();
+list_of_file_names.pop();
+getSingleFile(*session, file_name);
+  }
+}
+
+void GetFile::getSingleFile(core::ProcessSession& session, const std::string& 
file_name) const {
+  logger_->log_info("GetFile process %s", file_name);
+  auto flow_file = session.create();
+  gsl_Expects(flow_file);
+  std::string path;
+  std::string name;
+  std::tie(path, name) = utils::file::split_path(file_name);
+  flow_file->setAttribute(core::SpecialFlowAttribute::FILENAME, name);
+  flow_file->setAttribute(core::SpecialFlowAttribute::PATH, path);
+  flow_file->addAttribute(core::SpecialFlowAttribute::ABSOLUTE_PATH, 
file_name);
+
+  try {
+utils::FileReaderCallback file_reader_callback{file_name};
+session.write(flow_file, &file_reader_callback);
+session.transfer(flow_file, Success);
+if (!request_.keepSourceFile) {
+  auto remove_status = remove(file_name.c_str());
+  if (remove_status != 0) {
+logger_->log_error("GetFile could not delete file '%s', error %d: %s", 
file_name, errno, strerror(errno));
   }
-} catch (std::exception &exception) {
-  logger_->log_debug("GetFile Caught Exception %s", exception.what());
-  throw;
-} catch (...) {
-  throw;
 }
+  } catch (const std::exception& exception) {
+logger_->log_error("GetFile caught exception while processing file '%s': 
%s", file_name, exception.what());
+flow_file->setDeleted(true);

Review comment:
   Okay, I need more coffee, didn't realise that this is a different 
scenario. :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1064: MINIFICPP-987 Improve error handling in GetFile

[GitBox]

fgerlits commented on a change in pull request #1064:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1064#discussion_r630912957



##
File path: extensions/standard-processors/processors/GetFile.cpp
##
@@ -158,76 +147,89 @@ void GetFile::onSchedule(core::ProcessContext *context, 
core::ProcessSessionFact
   request_.inputDirectory = value;
 }
 
-void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession *session) {
-  // Perform directory list
-
+void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession* session) {
   metrics_->iterations_++;
 
-  const bool isDirEmptyBeforePoll = isListingEmpty();
-  logger_->log_debug("Is listing empty before polling directory %i", 
isDirEmptyBeforePoll);
-  if (isDirEmptyBeforePoll) {
+  const bool is_dir_empty_before_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s before polling directory", 
is_dir_empty_before_poll ? "empty" : "not empty");

Review comment:
   I'll change it if you think saving 5 chars in the line (and 5 bytes in 
the code) is worth it, but I think it is more readable this way.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1064: MINIFICPP-987 Improve error handling in GetFile

[GitBox]

fgerlits commented on a change in pull request #1064:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1064#discussion_r630911063



##
File path: extensions/standard-processors/processors/GetFile.cpp
##
@@ -158,76 +147,89 @@ void GetFile::onSchedule(core::ProcessContext *context, 
core::ProcessSessionFact
   request_.inputDirectory = value;
 }
 
-void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession *session) {
-  // Perform directory list
-
+void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession* session) {
   metrics_->iterations_++;
 
-  const bool isDirEmptyBeforePoll = isListingEmpty();
-  logger_->log_debug("Is listing empty before polling directory %i", 
isDirEmptyBeforePoll);
-  if (isDirEmptyBeforePoll) {
+  const bool is_dir_empty_before_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s before polling directory", 
is_dir_empty_before_poll ? "empty" : "not empty");
+  if (is_dir_empty_before_poll) {
 if (request_.pollInterval == 0 || (utils::timeutils::getTimeMillis() - 
last_listing_time_) > request_.pollInterval) {
   performListing(request_);
   last_listing_time_.store(utils::timeutils::getTimeMillis());
 }
   }
 
-  const bool isDirEmptyAfterPoll = isListingEmpty();
-  logger_->log_debug("Is listing empty after polling directory %i", 
isDirEmptyAfterPoll);
-
-  if (!isDirEmptyAfterPoll) {
-try {
-  std::queue list;
-  pollListing(list, request_);
-  while (!list.empty()) {
-std::string fileName = list.front();
-list.pop();
-logger_->log_info("GetFile process %s", fileName);
-auto flowFile = session->create();
-if (flowFile == nullptr)
-  return;
-std::size_t found = fileName.find_last_of("/\\");
-std::string path = fileName.substr(0, found);
-std::string name = fileName.substr(found + 1);
-flowFile->setAttribute(core::SpecialFlowAttribute::FILENAME, name);
-flowFile->setAttribute(core::SpecialFlowAttribute::PATH, path);
-flowFile->addAttribute(core::SpecialFlowAttribute::ABSOLUTE_PATH, 
fileName);
-session->import(fileName, flowFile, request_.keepSourceFile);
-session->transfer(flowFile, Success);
+  const bool is_dir_empty_after_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s after polling directory", 
is_dir_empty_after_poll ? "empty" : "not empty");
+  if (is_dir_empty_after_poll) {
+yield();
+return;
+  }
+
+  std::queue list_of_file_names = pollListing(request_.batchSize);
+  while (!list_of_file_names.empty()) {
+std::string file_name = list_of_file_names.front();
+list_of_file_names.pop();
+getSingleFile(*session, file_name);
+  }
+}
+
+void GetFile::getSingleFile(core::ProcessSession& session, const std::string& 
file_name) const {
+  logger_->log_info("GetFile process %s", file_name);
+  auto flow_file = session.create();
+  gsl_Expects(flow_file);
+  std::string path;
+  std::string name;
+  std::tie(path, name) = utils::file::split_path(file_name);
+  flow_file->setAttribute(core::SpecialFlowAttribute::FILENAME, name);
+  flow_file->setAttribute(core::SpecialFlowAttribute::PATH, path);
+  flow_file->addAttribute(core::SpecialFlowAttribute::ABSOLUTE_PATH, 
file_name);
+
+  try {
+utils::FileReaderCallback file_reader_callback{file_name};
+session.write(flow_file, &file_reader_callback);
+session.transfer(flow_file, Success);
+if (!request_.keepSourceFile) {
+  auto remove_status = remove(file_name.c_str());
+  if (remove_status != 0) {
+logger_->log_error("GetFile could not delete file '%s', error %d: %s", 
file_name, errno, strerror(errno));
   }
-} catch (std::exception &exception) {
-  logger_->log_debug("GetFile Caught Exception %s", exception.what());
-  throw;
-} catch (...) {
-  throw;
 }
+  } catch (const std::exception& exception) {
+logger_->log_error("GetFile caught exception while processing file '%s': 
%s", file_name, exception.what());
+flow_file->setDeleted(true);

Review comment:
   We don't want to roll back if reading one file failed but another file 
succeeded; in this case, we want to ignore the failed file but process the 
successful one.
   
   My understanding is that every flow file we create with `session->create()` 
has to either be transferred to an output relationship or marked as deleted, 
otherwise `routeFlowFile(flow_file)`, and therefore `session->commit()`, will 
throw.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] gschaer opened a new pull request #5069: NIFI-6685: Add align and distribute UI actions

[GitBox]

gschaer opened a new pull request #5069:
URL: https://github.com/apache/nifi/pull/5069


   
   Thank you for submitting a contribution to Apache NiFi.
   
   Please provide a short description of the PR here:
   
    Description of PR
   
   Enables align and distribute actions on the core UI. When at least two 
components are selected, the user can now align them vertically or 
horizontally. When at least three components are selected, the user can now 
distribute them evenly.
   
   Carbon Icons library was added to illustrate the actions in the UI. (Apache 
License 2.0)
   This fixes bug NIFI-6685
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [X] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [X] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [X] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [X] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [X] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [X] Have you verified that the full build is successful on JDK 8?
   - [X] Have you verified that the full build is successful on JDK 11?
   - [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1070: MINIFICPP-1556 Fix setting partial or invalid credentials in AWS credentials service

[GitBox]

adamdebreceni commented on a change in pull request #1070:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1070#discussion_r630897852



##
File path: libminifi/test/aws-tests/PutS3ObjectTests.cpp
##
@@ -107,6 +107,17 @@ TEST_CASE_METHOD(PutS3ObjectTestsFixture, "Test required 
property not set", "[aw
   REQUIRE_THROWS_AS(test_controller.runSession(plan, true), 
minifi::Exception&);
 }
 
+TEST_CASE_METHOD(PutS3ObjectTestsFixture, "Test incomplete credentials in 
credentials service", "[awsS3Config]") {
+  setBucket();
+  plan->setProperty(aws_credentials_service, "Secret Key", "secret");
+  setCredentialsService();
+  REQUIRE_THROWS_AS(test_controller.runSession(plan, true), 
minifi::Exception&);
+  REQUIRE(verifyLogLinePresenceInPollTime(std::chrono::seconds(3), "AWS 
Credentials have not been set!"));
+
+  // Test that no invalid credentials file was set from previous properties
+  REQUIRE(!LogTestController::getInstance().contains("load configure file 
failed", std::chrono::seconds(0), std::chrono::milliseconds(0)));

Review comment:
   `REQUIRE_FALSE` might be used here, and two other places




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1070: MINIFICPP-1556 Fix setting partial or invalid credentials in AWS credentials service

[GitBox]

adamdebreceni commented on a change in pull request #1070:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1070#discussion_r630897852



##
File path: libminifi/test/aws-tests/PutS3ObjectTests.cpp
##
@@ -107,6 +107,17 @@ TEST_CASE_METHOD(PutS3ObjectTestsFixture, "Test required 
property not set", "[aw
   REQUIRE_THROWS_AS(test_controller.runSession(plan, true), 
minifi::Exception&);
 }
 
+TEST_CASE_METHOD(PutS3ObjectTestsFixture, "Test incomplete credentials in 
credentials service", "[awsS3Config]") {
+  setBucket();
+  plan->setProperty(aws_credentials_service, "Secret Key", "secret");
+  setCredentialsService();
+  REQUIRE_THROWS_AS(test_controller.runSession(plan, true), 
minifi::Exception&);
+  REQUIRE(verifyLogLinePresenceInPollTime(std::chrono::seconds(3), "AWS 
Credentials have not been set!"));
+
+  // Test that no invalid credentials file was set from previous properties
+  REQUIRE(!LogTestController::getInstance().contains("load configure file 
failed", std::chrono::seconds(0), std::chrono::milliseconds(0)));

Review comment:
   `REQUIRE_FALSE` might be used here




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni edited a comment on pull request #1057: MINIFICPP-1537 - Log heartbeats on demand

[GitBox]

adamdebreceni edited a comment on pull request #1057:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1057#issuecomment-839636962


   > Looks good to me, however I think we could also add some tests as well, 
maybe something along the lines of IntegrationBase where you set the 
heartbeat.reporter.classes property and then verify the logs with 
verifyLogLinePresenceInPollTime
   
   added test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on pull request #1057: MINIFICPP-1537 - Log heartbeats on demand

[GitBox]

adamdebreceni commented on pull request #1057:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1057#issuecomment-839636962


   > Looks good to me, however I think we could also add some tests as well, 
maybe something along the lines of IntegrationBase where you set the 
heartbeat.reporter.classes property and then verify the logs with 
verifyLogLinePresenceInPollTime
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni closed pull request #1057: MINIFICPP-1537 - Log heartbeats on demand

[GitBox]

adamdebreceni closed pull request #1057:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1057


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #1064: MINIFICPP-987 Improve error handling in GetFile

[GitBox]

arpadboda commented on a change in pull request #1064:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1064#discussion_r630885656



##
File path: extensions/standard-processors/processors/GetFile.cpp
##
@@ -158,76 +147,89 @@ void GetFile::onSchedule(core::ProcessContext *context, 
core::ProcessSessionFact
   request_.inputDirectory = value;
 }
 
-void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession *session) {
-  // Perform directory list
-
+void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession* session) {
   metrics_->iterations_++;
 
-  const bool isDirEmptyBeforePoll = isListingEmpty();
-  logger_->log_debug("Is listing empty before polling directory %i", 
isDirEmptyBeforePoll);
-  if (isDirEmptyBeforePoll) {
+  const bool is_dir_empty_before_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s before polling directory", 
is_dir_empty_before_poll ? "empty" : "not empty");
+  if (is_dir_empty_before_poll) {
 if (request_.pollInterval == 0 || (utils::timeutils::getTimeMillis() - 
last_listing_time_) > request_.pollInterval) {
   performListing(request_);
   last_listing_time_.store(utils::timeutils::getTimeMillis());
 }
   }
 
-  const bool isDirEmptyAfterPoll = isListingEmpty();
-  logger_->log_debug("Is listing empty after polling directory %i", 
isDirEmptyAfterPoll);
-
-  if (!isDirEmptyAfterPoll) {
-try {
-  std::queue list;
-  pollListing(list, request_);
-  while (!list.empty()) {
-std::string fileName = list.front();
-list.pop();
-logger_->log_info("GetFile process %s", fileName);
-auto flowFile = session->create();
-if (flowFile == nullptr)
-  return;
-std::size_t found = fileName.find_last_of("/\\");
-std::string path = fileName.substr(0, found);
-std::string name = fileName.substr(found + 1);
-flowFile->setAttribute(core::SpecialFlowAttribute::FILENAME, name);
-flowFile->setAttribute(core::SpecialFlowAttribute::PATH, path);
-flowFile->addAttribute(core::SpecialFlowAttribute::ABSOLUTE_PATH, 
fileName);
-session->import(fileName, flowFile, request_.keepSourceFile);
-session->transfer(flowFile, Success);
+  const bool is_dir_empty_after_poll = isListingEmpty();
+  logger_->log_debug("Listing is %s after polling directory", 
is_dir_empty_after_poll ? "empty" : "not empty");
+  if (is_dir_empty_after_poll) {
+yield();
+return;
+  }
+
+  std::queue list_of_file_names = pollListing(request_.batchSize);
+  while (!list_of_file_names.empty()) {
+std::string file_name = list_of_file_names.front();
+list_of_file_names.pop();
+getSingleFile(*session, file_name);
+  }
+}
+
+void GetFile::getSingleFile(core::ProcessSession& session, const std::string& 
file_name) const {
+  logger_->log_info("GetFile process %s", file_name);
+  auto flow_file = session.create();
+  gsl_Expects(flow_file);
+  std::string path;
+  std::string name;
+  std::tie(path, name) = utils::file::split_path(file_name);
+  flow_file->setAttribute(core::SpecialFlowAttribute::FILENAME, name);
+  flow_file->setAttribute(core::SpecialFlowAttribute::PATH, path);
+  flow_file->addAttribute(core::SpecialFlowAttribute::ABSOLUTE_PATH, 
file_name);
+
+  try {
+utils::FileReaderCallback file_reader_callback{file_name};
+session.write(flow_file, &file_reader_callback);
+session.transfer(flow_file, Success);
+if (!request_.keepSourceFile) {
+  auto remove_status = remove(file_name.c_str());
+  if (remove_status != 0) {
+logger_->log_error("GetFile could not delete file '%s', error %d: %s", 
file_name, errno, strerror(errno));
   }
-} catch (std::exception &exception) {
-  logger_->log_debug("GetFile Caught Exception %s", exception.what());
-  throw;
-} catch (...) {
-  throw;
 }
+  } catch (const std::exception& exception) {
+logger_->log_error("GetFile caught exception while processing file '%s': 
%s", file_name, exception.what());
+flow_file->setDeleted(true);

Review comment:
   Do we need this? 
   I would expect session->rollback() later to clean up all the "mess" we made 
in an interrupted onTrigger

##
File path: extensions/standard-processors/processors/GetFile.cpp
##
@@ -158,76 +147,89 @@ void GetFile::onSchedule(core::ProcessContext *context, 
core::ProcessSessionFact
   request_.inputDirectory = value;
 }
 
-void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession *session) {
-  // Perform directory list
-
+void GetFile::onTrigger(core::ProcessContext* /*context*/, 
core::ProcessSession* session) {
   metrics_->iterations_++;
 
-  const bool isDirEmptyBeforePoll = isListingEmpty();
-  logger_->log_debug("Is listing empty before polling directory %i", 
isDirEmptyBeforePoll);
-  if (isDirEmptyBeforePoll) {
+  const bool is_dir_empty_before_poll = isListingEmpty();
+  logger_->log_debug(

[GitHub] [nifi-minifi-cpp] aminadinari19 commented on a change in pull request #1040: MINIFICPP-1329- Fix implementation and usages of string to bool

[GitBox]

aminadinari19 commented on a change in pull request #1040:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1040#discussion_r630852673



##
File path: controller/MiNiFiController.cpp
##
@@ -167,9 +165,10 @@ int main(int argc, char **argv) {
   socket = secure_context != nullptr ? 
stream_factory_->createSecureSocket(host, port, secure_context) : 
stream_factory_->createSocket(host, port);
   if (getConnectionSize(std::move(socket), std::cout, connection) < 0)
 std::cout << "Could not connect to remote host " << host << ":" << 
port << std::endl;
-} else
+} else {
   std::cout << "Could not connect to remote host " << host << ":" << 
port << std::endl;
   }
+  }

Review comment:
   Sorry I thought this is an old comment and didn't see the date. 
Hopefully it's correct now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1057: MINIFICPP-1537 - Log heartbeats on demand

[GitBox]

adamdebreceni commented on a change in pull request #1057:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1057#discussion_r630823120



##
File path: conf/minifi.properties
##
@@ -67,6 +67,7 @@ nifi.content.repository.class.name=DatabaseContentRepository
 ## define protocol parameters
 ## The default is CoAP, if that extension is built. 
 ## Alternatively, you may use RESTSender if http-curl is built
+#nifi.c2.agent.protocol.class=RESTSender

Review comment:
   it became obsolete in light of #1058, removed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-8532) ConsumeKinesisStream tuning/performance testing

[Peter Turcsanyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Turcsanyi updated NIFI-8532:
--
Description: 
{{ConsumeKinesisStream}} was added in NIFI-2892 with some seemingly sensible 
default property settings that are passed to the Kinesis Client Library.

Some performance testing/experimentation should be conducted for different NiFi 
setups (e.g. single instance, remote instance, cluster) to determine whether 
other settings should be used by default.

For example (from comments in PR 
[#4822|https://github.com/apache/nifi/pull/4822]):
{quote}
[~turcsanyip] did some kind of performance tests with 1 Mio messages in 12 
shards and running NiFi on [his] local machine (no cluster). It took 64-74 
seconds with default settings (no additional dynamic properties).

The interesting thing that [he] could not see any difference when I set 
Checkpoint Interval to 0 sec (that is checkpointing "synchronously" after each 
bunch of messages received in IRecordProcessor.processRecords() callback). It 
seems there is no significant overhead of checkpointing more frequently (and it 
has the advantage of having fewer duplicated messages in case of restart).

It can be investigated further in a follow-up ticket with more sophisticated 
performance tests (nifi cluster, non-local machine, tuning KCL properties, etc) 
and the default can be adjusted if that is reasonable.
{quote}
(this is *that* ticket)

  was:
{{ConsumeKinesisStream}} was added in NIFI-2982 with some seemingly sensible 
default property settings that are passed to the Kinesis Client Library.

Some performance testing/experimentation should be conducted for different NiFi 
setups (e.g. single instance, remote instance, cluster) to determine whether 
other settings should be used by default.

For example (from comments in PR 
[#4822|https://github.com/apache/nifi/pull/4822]):
{quote}
[~turcsanyip] did some kind of performance tests with 1 Mio messages in 12 
shards and running NiFi on [his] local machine (no cluster). It took 64-74 
seconds with default settings (no additional dynamic properties).

The interesting thing that [he] could not see any difference when I set 
Checkpoint Interval to 0 sec (that is checkpointing "synchronously" after each 
bunch of messages received in IRecordProcessor.processRecords() callback). It 
seems there is no significant overhead of checkpointing more frequently (and it 
has the advantage of having fewer duplicated messages in case of restart).

It can be investigated further in a follow-up ticket with more sophisticated 
performance tests (nifi cluster, non-local machine, tuning KCL properties, etc) 
and the default can be adjusted if that is reasonable.
{quote}
(this is *that* ticket)


> ConsumeKinesisStream tuning/performance testing
> ---
>
> Key: NIFI-8532
> URL: https://issues.apache.org/jira/browse/NIFI-8532
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Chris Sampson
>Priority: Major
>
> {{ConsumeKinesisStream}} was added in NIFI-2892 with some seemingly sensible 
> default property settings that are passed to the Kinesis Client Library.
> Some performance testing/experimentation should be conducted for different 
> NiFi setups (e.g. single instance, remote instance, cluster) to determine 
> whether other settings should be used by default.
> For example (from comments in PR 
> [#4822|https://github.com/apache/nifi/pull/4822]):
> {quote}
> [~turcsanyip] did some kind of performance tests with 1 Mio messages in 12 
> shards and running NiFi on [his] local machine (no cluster). It took 64-74 
> seconds with default settings (no additional dynamic properties).
> The interesting thing that [he] could not see any difference when I set 
> Checkpoint Interval to 0 sec (that is checkpointing "synchronously" after 
> each bunch of messages received in IRecordProcessor.processRecords() 
> callback). It seems there is no significant overhead of checkpointing more 
> frequently (and it has the advantage of having fewer duplicated messages in 
> case of restart).
> It can be investigated further in a follow-up ticket with more sophisticated 
> performance tests (nifi cluster, non-local machine, tuning KCL properties, 
> etc) and the default can be adjusted if that is reasonable.
> {quote}
> (this is *that* ticket)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)