[GitHub] [nifi-minifi-cpp] aminadinari19 commented on a change in pull request #1040: MINIFICPP-1329- Fix implementation and usages of string to bool
aminadinari19 commented on a change in pull request #1040:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1040#discussion_r618944659
##
File path: extensions/standard-processors/processors/HashContent.cpp
##
@@ -84,6 +82,7 @@ void HashContent::onTrigger(core::ProcessContext *,
core::ProcessSession *sessio
}
if (failOnEmpty_ && flowFile->getSize() == 0) {
+logger_->log_trace("Failure as flow file is empty");
Review comment:
Okay. Done :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] aminadinari19 commented on a change in pull request #1040: MINIFICPP-1329- Fix implementation and usages of string to bool
aminadinari19 commented on a change in pull request #1040:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1040#discussion_r618943625
##
File path: libminifi/include/utils/StringUtils.h
##
@@ -75,13 +75,10 @@ struct string_traits{
class StringUtils {
public:
/**
- * Converts a string to a boolean
- * Better handles mixed case.
+ * Checks and converts a string to a boolean
* @param input input string
- * @param output output string.
+ * @returns an optional of a boolean: true if the string is "true" (ignoring
case), false if it is "false" (ignoring case), nullopt for any other value
*/
- static bool StringToBool(std::string input, bool );
Review comment:
Yeah I am not sure why I left that part. Anyway, fixed it now :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-8438) 2021-04-17 13:00:06,613 WARN [qtp771920473-7277] o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=3ea84d6d-042d-36bd-34e7-d7f47919c28d] Request from 172.30.28.10
[ https://issues.apache.org/jira/browse/NIFI-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329895#comment-17329895 ] Oscar commented on NIFI-8438: - 遇到這個問題，很嚴重，無解；目前只能通過不停的重啓解決 > 2021-04-17 13:00:06,613 WARN [qtp771920473-7277] > o.a.n.p.standard.HandleHttpRequest > HandleHttpRequest[id=3ea84d6d-042d-36bd-34e7-d7f47919c28d] Request from > 172.30.28.101 cannot be processed, container queue is full; responding with > SERVICE_UNAVAILABLE > > > Key: NIFI-8438 > URL: https://issues.apache.org/jira/browse/NIFI-8438 > Project: Apache NiFi > Issue Type: Bug > Components: NiFi Stateless >Affects Versions: 1.12.1 > Environment: CentOS Linux release 7.3.1611 (Core) nifi-1.12.1 >Reporter: YE WANG >Priority: Blocker > Labels: patch > Original Estimate: 96h > Remaining Estimate: 96h > > 频繁爆出 > Request from xxx cannot be processed, container queue is full; responding > with SERVICE_UNAVAILABLE.nifi 网站还能进去但是已经写不进任何资料。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] MikeThomsen commented on pull request #4646: NIFI-6047 Add DeduplicateRecords (combines 6047 and 6014)
MikeThomsen commented on pull request #4646: URL: https://github.com/apache/nifi/pull/4646#issuecomment-825221203 Thanks @ottobackwards. I'll try to make some time to knock these out tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] attilabukor commented on pull request #5023: NIFI-8463: Support custom SASL protocol name in PutKudu
attilabukor commented on pull request #5023: URL: https://github.com/apache/nifi/pull/5023#issuecomment-825210309 cc @granthenke -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] gardellajuanpablo commented on a change in pull request #5024: NIFI-8320: Fix column mismatch in PutDatabaseRecord
gardellajuanpablo commented on a change in pull request #5024:
URL: https://github.com/apache/nifi/pull/5024#discussion_r618745085
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java
##
@@ -691,14 +691,14 @@ private void executeDML(final ProcessContext context,
final ProcessSession sessi
final Object[] values = currentRecord.getValues();
final List dataTypes =
currentRecord.getSchema().getDataTypes();
-List columns =
tableSchema.getColumnsAsList();
for (int i = 0; i < fieldIndexes.size(); i++) {
final int currentFieldIndex = fieldIndexes.get(i);
Object currentValue = values[currentFieldIndex];
final DataType dataType =
dataTypes.get(currentFieldIndex);
final int fieldSqlType =
DataTypeUtils.getSQLTypeValue(dataType);
-final ColumnDescription column =
columns.get(currentFieldIndex);
+String columnName =
normalizeColumnName(currentRecord.getSchema().getField(currentFieldIndex).getFieldName(),
settings.translateFieldNames);
Review comment:
Is it required NPE/IOE checks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-8467) Improve UX of adding a process group using a flow definition
[
https://issues.apache.org/jira/browse/NIFI-8467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew M. Lim updated NIFI-8467:
Summary: Improve UX of adding a process group using a flow definition
(was: Improve UX of Add Process Group file upload)
> Improve UX of adding a process group using a flow definition
>
>
> Key: NIFI-8467
> URL: https://issues.apache.org/jira/browse/NIFI-8467
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core UI
>Affects Versions: 1.14.0
>Reporter: Andrew M. Lim
>Priority: Major
> Attachments: import2.png
>
>
> https://issues.apache.org/jira/browse/NIFI-8260 added the ability to select a
> flow definition file to add a new process group to the NiFi canvas. This is
> a very useful feature, but I had some thoughts on how to improve the UX:
> * The placeholder text says "Enter a name or select a file to upload" which
> is ambiguous and doesn't exactly describe what is happening when using this
> feature.
> I'm assuming that the only valid files to select are flow definitions (JSON
> file). So it would be helpful to say "select a flow definition" in the
> placeholder text instead. Also, saying "upload" can be interpreted
> differently. If something is uploaded, I assume it is stored somewhere in the
> application and can be selected later for future use. That is consistent with
> the "Upload Template" functionality we have in NiFi. You upload a template
> first. Then, you can add that template to the canvas. So this would be my
> suggested new placeholder text if references to "upload" are removed:
> "Enter a name or select a flow definition file"
> Note: If this change is implemented, the text "File to Upload:" that is shown
> after selecting the file should be changed to "Flow definition file:"
> * The icon added in the Process Group Name field is the same icon used for
> "Upload Template" in the Operate Palette. It would be helpful if we had a
> different icon to avoid confusion.
> * This is not related specifically to the change from NIFI-8260, but related
> to the dialog under discussion. The "Import from Registry" icon seems to
> signify a download (from the cloud) not an import. Perhaps replace with the
> icon I'm attaching to this Jira where the arrow is sideways.
> Looking beyond the above minor tweaks to the UI, here are some thoughts that
> may require larger changes to address/implement:
> * I like how the new feature allows the user to choose the name of the new
> process group when a flow definition file is selected. But, I found it
> confusing when the user doesn't enter a name. In this case, the name defaults
> to the name of the file itself. I expected the name to default to the
> process group name defined _within_ the JSON file. For example:
> {
> "flowContents": {
> "identifier": "dca9377b-b785-34fe-b594-194f526aa914",
> "name": "PG_Name",
> "comments": "",
> "position":
> { "x": 535, "y": 284 }
> ,
> When the same flow definition JSON file is imported into Registry. If you
> then import this from the Registry, the name of the process group will be
> "PG_Name".
> * I feel like this new feature is more similar to "Import from Registry" and
> could therefore be implemented as a "Import from Flow Definition" link which
> opens a new Import from Flow Definition dialog. Details for this
> implementation would need to be fleshed out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8467) Improve UX of Add Process Group file upload
[
https://issues.apache.org/jira/browse/NIFI-8467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew M. Lim updated NIFI-8467:
Description:
https://issues.apache.org/jira/browse/NIFI-8260 added the ability to select a
flow definition file to add a new process group to the NiFi canvas. This is a
very useful feature, but I had some thoughts on how to improve the UX:
* The placeholder text says "Enter a name or select a file to upload" which is
ambiguous and doesn't exactly describe what is happening when using this
feature.
I'm assuming that the only valid files to select are flow definitions (JSON
file). So it would be helpful to say "select a flow definition" in the
placeholder text instead. Also, saying "upload" can be interpreted differently.
If something is uploaded, I assume it is stored somewhere in the application
and can be selected later for future use. That is consistent with the "Upload
Template" functionality we have in NiFi. You upload a template first. Then, you
can add that template to the canvas. So this would be my suggested new
placeholder text if references to "upload" are removed:
"Enter a name or select a flow definition file"
Note: If this change is implemented, the text "File to Upload:" that is shown
after selecting the file should be changed to "Flow definition file:"
* The icon added in the Process Group Name field is the same icon used for
"Upload Template" in the Operate Palette. It would be helpful if we had a
different icon to avoid confusion.
* This is not related specifically to the change from NIFI-8260, but related
to the dialog under discussion. The "Import from Registry" icon seems to
signify a download (from the cloud) not an import. Perhaps replace with the
icon I'm attaching to this Jira where the arrow is sideways.
Looking beyond the above minor tweaks to the UI, here are some thoughts that
may require larger changes to address/implement:
* I like how the new feature allows the user to choose the name of the new
process group when a flow definition file is selected. But, I found it
confusing when the user doesn't enter a name. In this case, the name defaults
to the name of the file itself. I expected the name to default to the process
group name defined _within_ the JSON file. For example:
{
"flowContents": {
"identifier": "dca9377b-b785-34fe-b594-194f526aa914",
"name": "PG_Name",
"comments": "",
"position":
{ "x": 535, "y": 284 }
,
When the same flow definition JSON file is imported into Registry. If you then
import this from the Registry, the name of the process group will be "PG_Name".
* I feel like this new feature is more similar to "Import from Registry" and
could therefore be implemented as a "Import from Flow Definition" link which
opens a new Import from Flow Definition dialog. Details for this implementation
would need to be fleshed out.
was:
https://issues.apache.org/jira/browse/NIFI-8260 added the ability to select a
flow definition file to add a new process group to the NiFi canvas. This is a
very useful feature, but I had some thoughts on how to improve the UX:
* The placeholder text says "Enter a name or select a file to upload" which is
ambiguous and doesn't exactly describe what is happening when using this
feature.
I'm assuming that the only valid files to select are flow definitions (JSON
file). So it would be helpful to say "select a flow definition" in the
placeholder text instead. Also, saying "upload" doesn't really describe the
action being taken. If something is uploaded, I assume it is stored somewhere
in the application and can be selected later for future use. That is consistent
with the "Upload Template" functionality we have in NiFi. You upload a template
first. Then, you can add that template to the canvas. So this would be my
suggested new placeholder text if references to "upload" are removed:
"Enter a name or select a flow definition file"
Note: If this change is implemented, the text "File to Upload:" that is shown
after selecting the file should be changed to "Flow definition file:"
* The icon added in the Process Group Name field is the same icon used for
"Upload Template" in the Operate Palette. It would be helpful if we had a
different icon to avoid confusion.
* This is not related specifically to the change from NIFI-8260, but related
to this dialog, the "Import from Registry" icon seems to signify a download
(from the cloud) not an import. Perhaps replace with the icon I'm attaching to
this Jira where the arrow is sideways.
Looking beyond the above minor tweaks to the UI, here are some thoughts that
may require larger changes to address/implement
* I like how the new feature allows the user to choose the name of the new
process group when a flow definition file is selected. But, I found it
confusing when the user doesn't enter a name. In this case, the name defaults
to the
[jira] [Updated] (NIFI-8467) Improve UX of Add Process Group file upload
[
https://issues.apache.org/jira/browse/NIFI-8467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew M. Lim updated NIFI-8467:
Description:
https://issues.apache.org/jira/browse/NIFI-8260 added the ability to select a
flow definition file to add a new process group to the NiFi canvas. This is a
very useful feature, but I had some thoughts on how to improve the UX:
* The placeholder text says "Enter a name or select a file to upload" which is
ambiguous and doesn't exactly describe what is happening when using this
feature.
I'm assuming that the only valid files to select are flow definitions (JSON
file). So it would be helpful to say "select a flow definition" in the
placeholder text instead. Also, saying "upload" doesn't really describe the
action being taken. If something is uploaded, I assume it is stored somewhere
in the application and can be selected later for future use. That is consistent
with the "Upload Template" functionality we have in NiFi. You upload a template
first. Then, you can add that template to the canvas. So this would be my
suggested new placeholder text if references to "upload" are removed:
"Enter a name or select a flow definition file"
Note: If this change is implemented, the text "File to Upload:" that is shown
after selecting the file should be changed to "Flow definition file:"
* The icon added in the Process Group Name field is the same icon used for
"Upload Template" in the Operate Palette. It would be helpful if we had a
different icon to avoid confusion.
* This is not related specifically to the change from NIFI-8260, but related
to this dialog, the "Import from Registry" icon seems to signify a download
(from the cloud) not an import. Perhaps replace with the icon I'm attaching to
this Jira where the arrow is sideways.
Looking beyond the above minor tweaks to the UI, here are some thoughts that
may require larger changes to address/implement
* I like how the new feature allows the user to choose the name of the new
process group when a flow definition file is selected. But, I found it
confusing when the user doesn't enter a name. In this case, the name defaults
to the name of the file itself. I expected the name to default to the process
group name defined within the JSON file. For example:
{
"flowContents": {
"identifier": "dca9377b-b785-34fe-b594-194f526aa914",
"name": "PG_Name",
"comments": "",
"position":
{ "x": 535, "y": 284 }
,
When the same flow definition JSON file is imported into Registry. If you then
import this from the Registry, the name of the process group will be "PG_Name".
* I feel like this new feature is more similar to "Import from Registry" and
could therefore be implemented as a "Import from Flow Definition" link which
opens a new Import from Flow Definition dialog. Details for this implementation
would need to be fleshed out.
was:
https://issues.apache.org/jira/browse/NIFI-8260 added the ability to select a
flow definition file to add a new process group to the NiFi canvas. This is a
very useful feature, but I had some thoughts on how to improve the UX:
* The placeholder text says "Enter a name or select a file to upload" which is
ambiguous and doesn't really describe what is happening when using this feature.
I'm assuming that the only valid files to select are flow definitions (JSON
file). So it would be helpful to say "select a flow definition" in the
placeholder text instead. Also, saying "upload" doesn't really describe the
action being taken. If something is uploaded, I assume it is stored somewhere
in the application and can be selected later for future use. That is consistent
with the "Upload Template" functionality we have in NiFi. You upload a template
first. Then, you can add that template to the canvas. So this would be my
suggested new placeholder text if references to "upload" are removed:
"Enter a name or select a flow definition file"
Note: If this change is implemented, the text "File to Upload:" that is shown
after selecting the file should be changed to "Flow definition file:"
* The icon added in the Process Group Name field is the same icon used for
"Upload Template" in the Operate Palette. It would be helpful if we had a
different icon to avoid confusion.
* This is not related specifically to the change from NIFI-8260, but related
to this dialog, the "Import from Registry" icon seems to signify a download
(from the cloud) not an import. Perhaps replace with the icon I'm attaching to
this Jira where the arrow is sideways.
Looking beyond the above minor tweaks to the UI, here are some thoughts that
may require larger changes to address/implement
* I like how the new feature allows the user to choose the name of the new
process group when a flow definition file is selected. But, I found it
confusing when the user doesn't enter a name. In this case, the name defaults
to the name of
[jira] [Created] (NIFI-8467) Improve UX of Add Process Group file upload
Andrew M. Lim created NIFI-8467:
---
Summary: Improve UX of Add Process Group file upload
Key: NIFI-8467
URL: https://issues.apache.org/jira/browse/NIFI-8467
Project: Apache NiFi
Issue Type: Improvement
Components: Core UI
Affects Versions: 1.14.0
Reporter: Andrew M. Lim
Attachments: import2.png
https://issues.apache.org/jira/browse/NIFI-8260 added the ability to select a
flow definition file to add a new process group to the NiFi canvas. This is a
very useful feature, but I had some thoughts on how to improve the UX:
* The placeholder text says "Enter a name or select a file to upload" which is
ambiguous and doesn't really describe what is happening when using this feature.
I'm assuming that the only valid files to select are flow definitions (JSON
file). So it would be helpful to say "select a flow definition" in the
placeholder text instead. Also, saying "upload" doesn't really describe the
action being taken. If something is uploaded, I assume it is stored somewhere
in the application and can be selected later for future use. That is consistent
with the "Upload Template" functionality we have in NiFi. You upload a template
first. Then, you can add that template to the canvas. So this would be my
suggested new placeholder text if references to "upload" are removed:
"Enter a name or select a flow definition file"
Note: If this change is implemented, the text "File to Upload:" that is shown
after selecting the file should be changed to "Flow definition file:"
* The icon added in the Process Group Name field is the same icon used for
"Upload Template" in the Operate Palette. It would be helpful if we had a
different icon to avoid confusion.
* This is not related specifically to the change from NIFI-8260, but related
to this dialog, the "Import from Registry" icon seems to signify a download
(from the cloud) not an import. Perhaps replace with the icon I'm attaching to
this Jira where the arrow is sideways.
Looking beyond the above minor tweaks to the UI, here are some thoughts that
may require larger changes to address/implement
* I like how the new feature allows the user to choose the name of the new
process group when a flow definition file is selected. But, I found it
confusing when the user doesn't enter a name. In this case, the name defaults
to the name of the file itself. I expected the name to default to the process
group name defined within the JSON file. For example:
{
"flowContents": {
"identifier": "dca9377b-b785-34fe-b594-194f526aa914",
"name": "PG_Name",
"comments": "",
"position": {
"x": 535,
"y": 284
},
When the same flow definition JSON file is imported into Registry. If you then
import this from the Registry, the name of the process group will be "PG_Name".
* I feel like this new feature is more similar to "Import from Registry" and
could therefore be implemented as a "Import from Flow Definition" link which
opens a new Import from Flow Definition dialog. Details for this implementation
would need to be fleshed out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8467) Improve UX of Add Process Group file upload
[
https://issues.apache.org/jira/browse/NIFI-8467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew M. Lim updated NIFI-8467:
Attachment: import2.png
> Improve UX of Add Process Group file upload
> ---
>
> Key: NIFI-8467
> URL: https://issues.apache.org/jira/browse/NIFI-8467
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core UI
>Affects Versions: 1.14.0
>Reporter: Andrew M. Lim
>Priority: Major
> Attachments: import2.png
>
>
> https://issues.apache.org/jira/browse/NIFI-8260 added the ability to select a
> flow definition file to add a new process group to the NiFi canvas. This is
> a very useful feature, but I had some thoughts on how to improve the UX:
> * The placeholder text says "Enter a name or select a file to upload" which
> is ambiguous and doesn't really describe what is happening when using this
> feature.
> I'm assuming that the only valid files to select are flow definitions (JSON
> file). So it would be helpful to say "select a flow definition" in the
> placeholder text instead. Also, saying "upload" doesn't really describe the
> action being taken. If something is uploaded, I assume it is stored somewhere
> in the application and can be selected later for future use. That is
> consistent with the "Upload Template" functionality we have in NiFi. You
> upload a template first. Then, you can add that template to the canvas. So
> this would be my suggested new placeholder text if references to "upload" are
> removed:
> "Enter a name or select a flow definition file"
> Note: If this change is implemented, the text "File to Upload:" that is shown
> after selecting the file should be changed to "Flow definition file:"
> * The icon added in the Process Group Name field is the same icon used for
> "Upload Template" in the Operate Palette. It would be helpful if we had a
> different icon to avoid confusion.
> * This is not related specifically to the change from NIFI-8260, but related
> to this dialog, the "Import from Registry" icon seems to signify a download
> (from the cloud) not an import. Perhaps replace with the icon I'm attaching
> to this Jira where the arrow is sideways.
> Looking beyond the above minor tweaks to the UI, here are some thoughts that
> may require larger changes to address/implement
> * I like how the new feature allows the user to choose the name of the new
> process group when a flow definition file is selected. But, I found it
> confusing when the user doesn't enter a name. In this case, the name defaults
> to the name of the file itself. I expected the name to default to the
> process group name defined within the JSON file. For example:
> {
> "flowContents": {
> "identifier": "dca9377b-b785-34fe-b594-194f526aa914",
> "name": "PG_Name",
> "comments": "",
> "position": {
> "x": 535,
> "y": 284
> },
> When the same flow definition JSON file is imported into Registry. If you
> then import this from the Registry, the name of the process group will be
> "PG_Name".
> * I feel like this new feature is more similar to "Import from Registry" and
> could therefore be implemented as a "Import from Flow Definition" link which
> opens a new Import from Flow Definition dialog. Details for this
> implementation would need to be fleshed out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8320) Fetching wrong schema from PostgreSQL DB
[
https://issues.apache.org/jira/browse/NIFI-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-8320:
---
Fix Version/s: 1.14.0
Status: Patch Available (was: In Progress)
> Fetching wrong schema from PostgreSQL DB
>
>
> Key: NIFI-8320
> URL: https://issues.apache.org/jira/browse/NIFI-8320
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.0
> Environment: Nifi Docker, PostgreSQL 11.10
>Reporter: Augusto Dias
>Assignee: Matt Burgess
>Priority: Critical
> Fix For: 1.14.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I believe Nifi is incorrectly assuming the table schema.
> Nifi is using the column type from the column just before (n-1) to validate
> the value with the column type.
> In this case, the column is varchar and the column just before is smallint.
> Nifi accuses error because it can't cast "1x/dia" to short.
> By changing the type of column n-1 the problem solves.
> Do you think these two issues are related? Thanks.
> --- Nifi version ---
> Nifi 1.13.0
> 02/10/2021 19:15:44 BRT
> Tagged nifi-1.13.0-RC4
> From
> [{{3bc6a12}}|https://github.com/apache/nifi/commit/3bc6a122091214b33eee17a270163d7ca26e2a0c]
> on branch UNKNOWN
> -- JDBC PostgreSQL Version
> 42.2.19
> --- Nifi Interface Bulletin
> 18:54:55 BRT ERROR
> PutDatabaseRecord[id=2d23c61d-bfe8-351a-d7e5-84332341d827] Failed to put
> Records to database for
> StandardFlowFileRecord[uuid=2cbcd514-b0a5-4de7-b19d-79fa4891d39b,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1615382414416-4, container=default,
> section=4], offset=18303,
> length=980],offset=0,name=2cbcd514-b0a5-4de7-b19d-79fa4891d39b,size=718].
> Routing to failure.: For input string: "1x/dia"
> - Nifi logs/nifi-app.log
> 2021-03-10 18:54:55,175 ERROR [Timer-Driven Process Thread-7]
> o.a.n.p.standard.PutDatabaseRecord
> PutDatabaseRecord[id=2d23c61d-bfe8-351a-d7e5-84332341d827] Failed to put
> Records to database for
> StandardFlowFileRecord[uuid=2cbcd514-b0a5-4de7-b19d-79fa4891d39b,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1615382414416-4, container=default,
> section=4], offset=18303,
> length=980],offset=0,name=2cbcd514-b0a5-4de7-b19d-79fa4891d39b,size=718].
> Routing to failure.: java.lang.NumberFormatException: For input string:
> "1x/dia"
> java.lang.NumberFormatException: For input string: "1x/dia"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580)
> at java.lang.Short.parseShort(Short.java:118)
> at java.lang.Short.parseShort(Short.java:144)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.toShort(DataTypeUtils.java:1614)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:203)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:152)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:148)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.executeDML(PutDatabaseRecord.java:707)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.putToDatabase(PutDatabaseRecord.java:838)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1173)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> --- Create Table
> CREATE TABLE frequencia (
> fkhospital int2 NOT NULL DEFAULT '1'::smallint,
> fkfrequencia varchar(250) NOT NULL,
> nome varchar(250) NOT NULL,
> frequenciadia float4
[GitHub] [nifi] mattyb149 opened a new pull request #5024: NIFI-8320: Fix column mismatch in PutDatabaseRecord
mattyb149 opened a new pull request #5024:
URL: https://github.com/apache/nifi/pull/5024
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
There are two index mappings in PutDatabaseRecord, one for included record
fields and a missing one for table columns. The code erroneously used the
record field index which may not map to the same column index in the table.
This change does a name lookup to get the column corresponding to the record at
the field index, and uses that to determine data type.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-5335) PGP processor cannot handle multiple passworded keys in a pgp keyring.
[ https://issues.apache.org/jira/browse/NIFI-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-5335. Fix Version/s: 1.14.0 Resolution: Fixed Implemented in EncryptContentPGP and DecryptContentPGP with supporting Controller Services. > PGP processor cannot handle multiple passworded keys in a pgp keyring. > -- > > Key: NIFI-5335 > URL: https://issues.apache.org/jira/browse/NIFI-5335 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration Management >Affects Versions: 1.5.0 > Environment: RHEL >Reporter: Bill SAndman >Assignee: David Handermann >Priority: Major > Labels: encryption, pgp, security > Fix For: 1.14.0 > > Attachments: EncryptContent.JPG > > > The Encrypt Content processor can handle only one passworded PGP Secret key > on a given keyring. > To decrypt files from multiple data sources with different passworded PGP > Secret Keys, we are forced to create a separate key ring for each data > source, containing a single passworded PGP Secret Key. > It would be optimal to be able to point the decryption processor at a single > NIFI user keyring that had multiple data source Secret Keys, each passworded. > We would then be able to pass either the SEC ID (eg; 1024D/671D6CAF or the > UID (eg. dunnhumby - SMG - Prod) and the password for the given Secret Key. > Ideally the attached image would also include a Private Key Identifier to > distinguish which jey on the ring to use, that would match the Private > Keyring Passphrase. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-1694) EncryptContent processor should accept keyring file or individual key file for PGP encryption/decryption
[
https://issues.apache.org/jira/browse/NIFI-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-1694.
Fix Version/s: 1.14.0
Resolution: Fixed
Implemented in EncryptContentPGP and DecryptContentPGP with supporting
Controller Services.
> EncryptContent processor should accept keyring file or individual key file
> for PGP encryption/decryption
>
>
> Key: NIFI-1694
> URL: https://issues.apache.org/jira/browse/NIFI-1694
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 0.6.0, 0.5.1
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.14.0
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> As reported on the mailing lists [1], prior to {{0.5.1}}, users could provide
> the path to an individual PGP public key file (i.e.
> {{exportedPublicKey.asc}}) as the {{Public Keyring File}} property in the
> {{EncryptContent}} processor. With [NIFI-1324], the handling of the keyring
> files became more strict. While this follows the explicit naming of the
> property, some users may have individual keys rather than the keyring file.
> Add logic to support individual key handling for public and secret keys in
> {{EncryptContent}} processor via {{OpenPGPKeyBasedEncryptor}}.
> [1]
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201603.mbox/%3ccaf3pksz8cdis5pvjnnm_qwoctl6pohr2gw-udnt0h2nfse3...@mail.gmail.com%3e
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-7396) Add encryption metadata attributes for PGP encryption
[ https://issues.apache.org/jira/browse/NIFI-7396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-7396. Fix Version/s: 1.14.0 Resolution: Fixed Implemented in EncryptContentPGP and DecryptContentPGP. > Add encryption metadata attributes for PGP encryption > - > > Key: NIFI-7396 > URL: https://issues.apache.org/jira/browse/NIFI-7396 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.11.4 >Reporter: Andy LoPresto >Assignee: David Handermann >Priority: Major > Labels: attribute, encryption, metadata, pgp, security > Fix For: 1.14.0 > > > The PBE and keyed encryption modes now provide encryption metadata in the > outgoing flowfile attribute. The same work should be done for PGP encryption, > but depends on the PGP refactoring currently underway. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-6708) Improve DSA/El Gamal PGP compatibility issues
[
https://issues.apache.org/jira/browse/NIFI-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-6708.
Fix Version/s: 1.14.0
Resolution: Fixed
Implemented in EncryptContentPGP and DecryptContentPGP with supporting
Controller Services.
> Improve DSA/El Gamal PGP compatibility issues
> -
>
> Key: NIFI-6708
> URL: https://issues.apache.org/jira/browse/NIFI-6708
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Extensions
>Affects Versions: 1.9.2
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Major
> Labels: dsa, el-gamal, encryption, gpg, pgp, security
> Fix For: 1.14.0
>
>
> Some users have complained about an inability to use DSA GPG keys in the
> {{EncryptContent}} processor. While DSA keys are only used for signatures,
> they are frequently paired with El Gamal encryption keys in GnuPG. I am
> exploring the OpenPGP implementation via Bouncy Castle to see if there are
> improvements we can make.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-5346) Allow EncryptContent processor to specify PGP key as processor property
[
https://issues.apache.org/jira/browse/NIFI-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-5346.
Fix Version/s: 1.14.0
Resolution: Fixed
Implemented in EncryptContentPGP and DecryptContentPGP with supporting
Controller Services.
> Allow EncryptContent processor to specify PGP key as processor property
> ---
>
> Key: NIFI-5346
> URL: https://issues.apache.org/jira/browse/NIFI-5346
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Major
> Labels: encryption, pgp, security
> Fix For: 1.14.0
>
> Time Spent: 11h
> Remaining Estimate: 0h
>
> Users have requested the capability to paste the ASCII-armored key contents
> into an {{EncryptContent}} processor property in order to decouple from from
> an external keyring (for both encryption and decryption).
> The private key would be protected as a sensitive property (encrypted in the
> flow.xml.gz the same as a password field). The public key can either be
> protected in the same way, or treated as a plaintext value (it is not
> sensitive). There should be an additional field to record the unique
> identifier of the respective key (i.e. key ID or fingerprint + description).
> Specifying all of this information may be confusing on the default processor
> property dialog, and so an "Advanced"/"PGP" custom UI should be provided
> which organizes this information in a helpful way.
> Obviously, encrypting with a literal public key or decrypting with a literal
> private key should be easily interoperable with another encryption/decryption
> operation (either with another instance of {{EncryptContent}} within NiFi
> referencing an external keyring or using an external tool like GnuPG).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-2983) EncryptContent PGP mode can't handle signed encrypted files
[ https://issues.apache.org/jira/browse/NIFI-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-2983. Fix Version/s: 1.14.0 Resolution: Fixed Implemented in DecryptContentPGP. > EncryptContent PGP mode can't handle signed encrypted files > --- > > Key: NIFI-2983 > URL: https://issues.apache.org/jira/browse/NIFI-2983 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Environment: Apache NiFi 1.0.0 on Ubuntu 16.04 LTS. >Reporter: Emil Frank >Assignee: David Handermann >Priority: Minor > Fix For: 1.14.0 > > > The EncryptContent processor can't handle signed and encrypted PGP files - > just encrypted files work absolutely fine. > The error returned when an additionally signed file is (think it comes from > bouncycastle): > "encrypted message contains signed message - not literal data" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8251) Add Encrypt and Decrypt PGP Processors and Services
[ https://issues.apache.org/jira/browse/NIFI-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-8251: --- Fix Version/s: 1.14.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add Encrypt and Decrypt PGP Processors and Services > --- > > Key: NIFI-8251 > URL: https://issues.apache.org/jira/browse/NIFI-8251 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.13.0 >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Labels: encryption, gpg, pgp, security > Fix For: 1.14.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The purpose of this issue is to create new Processors and Controller Services > to perform encrypt and decrypt functions according to the OpenPGP > specification defined in [RFC 4880|https://tools.ietf.org/html/rfc4880]. This > functionality is implemented in the `EncryptContent` Processor, but the > current approach is difficult to configure and troubleshoot in some cases. > The new Controller Services should encapsulate management of keys and > keyrings to improve performance and provide better error handling. New > Controller Services would also support building additional Processors for > signing and verifying PGP messages as requested in NIFI-7322. Separating > encrypt and decrypt content functions into distinct Processors will provide a > clearer distinction between the properties necessary for configuration. > The implementation of these Processors and Controller Services should address > the issues described in the following issues: > - NIFI-7396 Add encryption metadata attributes > - NIFI-6708 Support DSA and ElGamal keys > - NIFI-5346 Specify PGP key as property > - NIFI-5335 Support multiple keys in a PGP keyring > - NIFI-2983 Support decryption of signed messages > - NIFI-1694 Support keyring or individual key for encryption or decryption -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8466) Offloading the Single Node load balancing target causes ignored backpressure
[ https://issues.apache.org/jira/browse/NIFI-8466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Gresock updated NIFI-8466: - Description: In a 3-node cluster, it is possible to get a Single Node load balanced queue into the following state by offloading the node that receives the load balanced flow files: # The load balanced queue ignores backpressure # The processor following the load balanced queue acts like it has nothing to process, even when running Steps to reproduce: # Add a Single Node load balanced queue on a multi-node cluster (see attached Template) # Discover which node receives the flow files in the queue by running only the first processor, then viewing the Connections tab of the Summary view, and finally clicking the Cluster button for the load balanced queue. The node that has all the flow files is the one you will need to Offload in the next step # Disconnect and then Offload the node that gets the flow files in Single Node load balancing # Reconnect the node # Run both processors surrounding the load balanced queue was: In a multi-node cluster, it is possible to get a Single Node load balanced queue into the following state by offloading the node that receives the load balanced flow files: # The load balanced queue ignores backpressure # The processor following the load balanced queue acts like it has nothing to process, even when running Steps to reproduce: # Add a Single Node load balanced queue on a multi-node cluster (see attached Template) # Discover which node receives the flow files in the queue by running only the first processor, then viewing the Connections tab of the Summary view, and finally clicking the Cluster button for the load balanced queue. The node that has all the flow files is the one you will need to Offload in the next step # Disconnect and then Offload the node that gets the flow files in Single Node load balancing # Reconnect the node # Run both processors surrounding the load balanced queue > Offloading the Single Node load balancing target causes ignored backpressure > > > Key: NIFI-8466 > URL: https://issues.apache.org/jira/browse/NIFI-8466 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Joseph Gresock >Priority: Minor > Attachments: Single_Node_Queue.xml > > > In a 3-node cluster, it is possible to get a Single Node load balanced queue > into the following state by offloading the node that receives the load > balanced flow files: > # The load balanced queue ignores backpressure > # The processor following the load balanced queue acts like it has nothing to > process, even when running > Steps to reproduce: > # Add a Single Node load balanced queue on a multi-node cluster (see attached > Template) > # Discover which node receives the flow files in the queue by running only > the first processor, then viewing the Connections tab of the Summary view, > and finally clicking the Cluster button for the load balanced queue. The > node that has all the flow files is the one you will need to Offload in the > next step > # Disconnect and then Offload the node that gets the flow files in Single > Node load balancing > # Reconnect the node > # Run both processors surrounding the load balanced queue -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8466) Offloading the Single Node load balancing target causes ignored backpressure
[ https://issues.apache.org/jira/browse/NIFI-8466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Gresock updated NIFI-8466: - Affects Version/s: (was: 1.13.2) 1.14.0 > Offloading the Single Node load balancing target causes ignored backpressure > > > Key: NIFI-8466 > URL: https://issues.apache.org/jira/browse/NIFI-8466 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Joseph Gresock >Priority: Minor > Attachments: Single_Node_Queue.xml > > > In a multi-node cluster, it is possible to get a Single Node load balanced > queue into the following state by offloading the node that receives the load > balanced flow files: > # The load balanced queue ignores backpressure > # The processor following the load balanced queue acts like it has nothing to > process, even when running > Steps to reproduce: > # Add a Single Node load balanced queue on a multi-node cluster (see attached > Template) > # Discover which node receives the flow files in the queue by running only > the first processor, then viewing the Connections tab of the Summary view, > and finally clicking the Cluster button for the load balanced queue. The > node that has all the flow files is the one you will need to Offload in the > next step > # Disconnect and then Offload the node that gets the flow files in Single > Node load balancing > # Reconnect the node > # Run both processors surrounding the load balanced queue -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8466) Offloading the Single Node load balancing target causes ignored backpressure
Joseph Gresock created NIFI-8466: Summary: Offloading the Single Node load balancing target causes ignored backpressure Key: NIFI-8466 URL: https://issues.apache.org/jira/browse/NIFI-8466 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.13.2 Reporter: Joseph Gresock Attachments: Single_Node_Queue.xml In a multi-node cluster, it is possible to get a Single Node load balanced queue into the following state by offloading the node that receives the load balanced flow files: # The load balanced queue ignores backpressure # The processor following the load balanced queue acts like it has nothing to process, even when running Steps to reproduce: # Add a Single Node load balanced queue on a multi-node cluster (see attached Template) # Discover which node receives the flow files in the queue by running only the first processor, then viewing the Connections tab of the Summary view, and finally clicking the Cluster button for the load balanced queue. The node that has all the flow files is the one you will need to Offload in the next step # Disconnect and then Offload the node that gets the flow files in Single Node load balancing # Reconnect the node # Run both processors surrounding the load balanced queue -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8251) Add Encrypt and Decrypt PGP Processors and Services
[ https://issues.apache.org/jira/browse/NIFI-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329386#comment-17329386 ] ASF subversion and git services commented on NIFI-8251: --- Commit 92bdc23adbe867c93f6e43c6665e84108d743fb2 in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=92bdc23 ] NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors - Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard implementations - NIFI-7396 EncryptContentPGP writes encryption metadata attributes - NIFI-6708 Controller Services support ElGamal Public and Private Keys - NIFI-5346 Controller Services support Keyring Files and ASCII Key properties - NIFI-5335 Controller Services support multiple public or private keys from keyrings - NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets regardless of signing - NIFI-1694 Controller Services support individual key files or keyrings NIFI-8251 Refactored Public Key ID Property to Public Key Search NIFI-8251 Corrected handling of multiple Encrypted Data packets in DecryptContentPGP - Added unit tests for encryption and decryption with both password-based and public key - Added PGP NAR dependencies to nifi-assembly Signed-off-by: Nathan Gough This closes #4842. > Add Encrypt and Decrypt PGP Processors and Services > --- > > Key: NIFI-8251 > URL: https://issues.apache.org/jira/browse/NIFI-8251 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.13.0 >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Labels: encryption, gpg, pgp, security > Time Spent: 2h 20m > Remaining Estimate: 0h > > The purpose of this issue is to create new Processors and Controller Services > to perform encrypt and decrypt functions according to the OpenPGP > specification defined in [RFC 4880|https://tools.ietf.org/html/rfc4880]. This > functionality is implemented in the `EncryptContent` Processor, but the > current approach is difficult to configure and troubleshoot in some cases. > The new Controller Services should encapsulate management of keys and > keyrings to improve performance and provide better error handling. New > Controller Services would also support building additional Processors for > signing and verifying PGP messages as requested in NIFI-7322. Separating > encrypt and decrypt content functions into distinct Processors will provide a > clearer distinction between the properties necessary for configuration. > The implementation of these Processors and Controller Services should address > the issues described in the following issues: > - NIFI-7396 Add encryption metadata attributes > - NIFI-6708 Support DSA and ElGamal keys > - NIFI-5346 Specify PGP key as property > - NIFI-5335 Support multiple keys in a PGP keyring > - NIFI-2983 Support decryption of signed messages > - NIFI-1694 Support keyring or individual key for encryption or decryption -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-1694) EncryptContent processor should accept keyring file or individual key file for PGP encryption/decryption
[
https://issues.apache.org/jira/browse/NIFI-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329384#comment-17329384
]
ASF subversion and git services commented on NIFI-1694:
---
Commit 92bdc23adbe867c93f6e43c6665e84108d743fb2 in nifi's branch
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=92bdc23 ]
NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors
- Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard
implementations
- NIFI-7396 EncryptContentPGP writes encryption metadata attributes
- NIFI-6708 Controller Services support ElGamal Public and Private Keys
- NIFI-5346 Controller Services support Keyring Files and ASCII Key properties
- NIFI-5335 Controller Services support multiple public or private keys from
keyrings
- NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets
regardless of signing
- NIFI-1694 Controller Services support individual key files or keyrings
NIFI-8251 Refactored Public Key ID Property to Public Key Search
NIFI-8251 Corrected handling of multiple Encrypted Data packets in
DecryptContentPGP
- Added unit tests for encryption and decryption with both password-based and
public key
- Added PGP NAR dependencies to nifi-assembly
Signed-off-by: Nathan Gough
This closes #4842.
> EncryptContent processor should accept keyring file or individual key file
> for PGP encryption/decryption
>
>
> Key: NIFI-1694
> URL: https://issues.apache.org/jira/browse/NIFI-1694
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 0.6.0, 0.5.1
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Minor
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> As reported on the mailing lists [1], prior to {{0.5.1}}, users could provide
> the path to an individual PGP public key file (i.e.
> {{exportedPublicKey.asc}}) as the {{Public Keyring File}} property in the
> {{EncryptContent}} processor. With [NIFI-1324], the handling of the keyring
> files became more strict. While this follows the explicit naming of the
> property, some users may have individual keys rather than the keyring file.
> Add logic to support individual key handling for public and secret keys in
> {{EncryptContent}} processor via {{OpenPGPKeyBasedEncryptor}}.
> [1]
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201603.mbox/%3ccaf3pksz8cdis5pvjnnm_qwoctl6pohr2gw-udnt0h2nfse3...@mail.gmail.com%3e
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-5335) PGP processor cannot handle multiple passworded keys in a pgp keyring.
[ https://issues.apache.org/jira/browse/NIFI-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329382#comment-17329382 ] ASF subversion and git services commented on NIFI-5335: --- Commit 92bdc23adbe867c93f6e43c6665e84108d743fb2 in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=92bdc23 ] NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors - Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard implementations - NIFI-7396 EncryptContentPGP writes encryption metadata attributes - NIFI-6708 Controller Services support ElGamal Public and Private Keys - NIFI-5346 Controller Services support Keyring Files and ASCII Key properties - NIFI-5335 Controller Services support multiple public or private keys from keyrings - NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets regardless of signing - NIFI-1694 Controller Services support individual key files or keyrings NIFI-8251 Refactored Public Key ID Property to Public Key Search NIFI-8251 Corrected handling of multiple Encrypted Data packets in DecryptContentPGP - Added unit tests for encryption and decryption with both password-based and public key - Added PGP NAR dependencies to nifi-assembly Signed-off-by: Nathan Gough This closes #4842. > PGP processor cannot handle multiple passworded keys in a pgp keyring. > -- > > Key: NIFI-5335 > URL: https://issues.apache.org/jira/browse/NIFI-5335 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration Management >Affects Versions: 1.5.0 > Environment: RHEL >Reporter: Bill SAndman >Assignee: David Handermann >Priority: Major > Labels: encryption, pgp, security > Attachments: EncryptContent.JPG > > > The Encrypt Content processor can handle only one passworded PGP Secret key > on a given keyring. > To decrypt files from multiple data sources with different passworded PGP > Secret Keys, we are forced to create a separate key ring for each data > source, containing a single passworded PGP Secret Key. > It would be optimal to be able to point the decryption processor at a single > NIFI user keyring that had multiple data source Secret Keys, each passworded. > We would then be able to pass either the SEC ID (eg; 1024D/671D6CAF or the > UID (eg. dunnhumby - SMG - Prod) and the password for the given Secret Key. > Ideally the attached image would also include a Private Key Identifier to > distinguish which jey on the ring to use, that would match the Private > Keyring Passphrase. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-5346) Allow EncryptContent processor to specify PGP key as processor property
[
https://issues.apache.org/jira/browse/NIFI-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329381#comment-17329381
]
ASF subversion and git services commented on NIFI-5346:
---
Commit 92bdc23adbe867c93f6e43c6665e84108d743fb2 in nifi's branch
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=92bdc23 ]
NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors
- Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard
implementations
- NIFI-7396 EncryptContentPGP writes encryption metadata attributes
- NIFI-6708 Controller Services support ElGamal Public and Private Keys
- NIFI-5346 Controller Services support Keyring Files and ASCII Key properties
- NIFI-5335 Controller Services support multiple public or private keys from
keyrings
- NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets
regardless of signing
- NIFI-1694 Controller Services support individual key files or keyrings
NIFI-8251 Refactored Public Key ID Property to Public Key Search
NIFI-8251 Corrected handling of multiple Encrypted Data packets in
DecryptContentPGP
- Added unit tests for encryption and decryption with both password-based and
public key
- Added PGP NAR dependencies to nifi-assembly
Signed-off-by: Nathan Gough
This closes #4842.
> Allow EncryptContent processor to specify PGP key as processor property
> ---
>
> Key: NIFI-5346
> URL: https://issues.apache.org/jira/browse/NIFI-5346
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Affects Versions: 1.7.0
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Major
> Labels: encryption, pgp, security
> Time Spent: 11h
> Remaining Estimate: 0h
>
> Users have requested the capability to paste the ASCII-armored key contents
> into an {{EncryptContent}} processor property in order to decouple from from
> an external keyring (for both encryption and decryption).
> The private key would be protected as a sensitive property (encrypted in the
> flow.xml.gz the same as a password field). The public key can either be
> protected in the same way, or treated as a plaintext value (it is not
> sensitive). There should be an additional field to record the unique
> identifier of the respective key (i.e. key ID or fingerprint + description).
> Specifying all of this information may be confusing on the default processor
> property dialog, and so an "Advanced"/"PGP" custom UI should be provided
> which organizes this information in a helpful way.
> Obviously, encrypting with a literal public key or decrypting with a literal
> private key should be easily interoperable with another encryption/decryption
> operation (either with another instance of {{EncryptContent}} within NiFi
> referencing an external keyring or using an external tool like GnuPG).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-6708) Improve DSA/El Gamal PGP compatibility issues
[
https://issues.apache.org/jira/browse/NIFI-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329380#comment-17329380
]
ASF subversion and git services commented on NIFI-6708:
---
Commit 92bdc23adbe867c93f6e43c6665e84108d743fb2 in nifi's branch
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=92bdc23 ]
NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors
- Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard
implementations
- NIFI-7396 EncryptContentPGP writes encryption metadata attributes
- NIFI-6708 Controller Services support ElGamal Public and Private Keys
- NIFI-5346 Controller Services support Keyring Files and ASCII Key properties
- NIFI-5335 Controller Services support multiple public or private keys from
keyrings
- NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets
regardless of signing
- NIFI-1694 Controller Services support individual key files or keyrings
NIFI-8251 Refactored Public Key ID Property to Public Key Search
NIFI-8251 Corrected handling of multiple Encrypted Data packets in
DecryptContentPGP
- Added unit tests for encryption and decryption with both password-based and
public key
- Added PGP NAR dependencies to nifi-assembly
Signed-off-by: Nathan Gough
This closes #4842.
> Improve DSA/El Gamal PGP compatibility issues
> -
>
> Key: NIFI-6708
> URL: https://issues.apache.org/jira/browse/NIFI-6708
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Extensions
>Affects Versions: 1.9.2
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Major
> Labels: dsa, el-gamal, encryption, gpg, pgp, security
>
> Some users have complained about an inability to use DSA GPG keys in the
> {{EncryptContent}} processor. While DSA keys are only used for signatures,
> they are frequently paired with El Gamal encryption keys in GnuPG. I am
> exploring the OpenPGP implementation via Bouncy Castle to see if there are
> improvements we can make.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7396) Add encryption metadata attributes for PGP encryption
[ https://issues.apache.org/jira/browse/NIFI-7396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329379#comment-17329379 ] ASF subversion and git services commented on NIFI-7396: --- Commit 92bdc23adbe867c93f6e43c6665e84108d743fb2 in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=92bdc23 ] NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors - Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard implementations - NIFI-7396 EncryptContentPGP writes encryption metadata attributes - NIFI-6708 Controller Services support ElGamal Public and Private Keys - NIFI-5346 Controller Services support Keyring Files and ASCII Key properties - NIFI-5335 Controller Services support multiple public or private keys from keyrings - NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets regardless of signing - NIFI-1694 Controller Services support individual key files or keyrings NIFI-8251 Refactored Public Key ID Property to Public Key Search NIFI-8251 Corrected handling of multiple Encrypted Data packets in DecryptContentPGP - Added unit tests for encryption and decryption with both password-based and public key - Added PGP NAR dependencies to nifi-assembly Signed-off-by: Nathan Gough This closes #4842. > Add encryption metadata attributes for PGP encryption > - > > Key: NIFI-7396 > URL: https://issues.apache.org/jira/browse/NIFI-7396 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.11.4 >Reporter: Andy LoPresto >Assignee: David Handermann >Priority: Major > Labels: attribute, encryption, metadata, pgp, security > > The PBE and keyed encryption modes now provide encryption metadata in the > outgoing flowfile attribute. The same work should be done for PGP encryption, > but depends on the PGP refactoring currently underway. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8251) Add Encrypt and Decrypt PGP Processors and Services
[ https://issues.apache.org/jira/browse/NIFI-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329378#comment-17329378 ] ASF subversion and git services commented on NIFI-8251: --- Commit 92bdc23adbe867c93f6e43c6665e84108d743fb2 in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=92bdc23 ] NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors - Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard implementations - NIFI-7396 EncryptContentPGP writes encryption metadata attributes - NIFI-6708 Controller Services support ElGamal Public and Private Keys - NIFI-5346 Controller Services support Keyring Files and ASCII Key properties - NIFI-5335 Controller Services support multiple public or private keys from keyrings - NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets regardless of signing - NIFI-1694 Controller Services support individual key files or keyrings NIFI-8251 Refactored Public Key ID Property to Public Key Search NIFI-8251 Corrected handling of multiple Encrypted Data packets in DecryptContentPGP - Added unit tests for encryption and decryption with both password-based and public key - Added PGP NAR dependencies to nifi-assembly Signed-off-by: Nathan Gough This closes #4842. > Add Encrypt and Decrypt PGP Processors and Services > --- > > Key: NIFI-8251 > URL: https://issues.apache.org/jira/browse/NIFI-8251 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.13.0 >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Labels: encryption, gpg, pgp, security > Time Spent: 2h 20m > Remaining Estimate: 0h > > The purpose of this issue is to create new Processors and Controller Services > to perform encrypt and decrypt functions according to the OpenPGP > specification defined in [RFC 4880|https://tools.ietf.org/html/rfc4880]. This > functionality is implemented in the `EncryptContent` Processor, but the > current approach is difficult to configure and troubleshoot in some cases. > The new Controller Services should encapsulate management of keys and > keyrings to improve performance and provide better error handling. New > Controller Services would also support building additional Processors for > signing and verifying PGP messages as requested in NIFI-7322. Separating > encrypt and decrypt content functions into distinct Processors will provide a > clearer distinction between the properties necessary for configuration. > The implementation of these Processors and Controller Services should address > the issues described in the following issues: > - NIFI-7396 Add encryption metadata attributes > - NIFI-6708 Support DSA and ElGamal keys > - NIFI-5346 Specify PGP key as property > - NIFI-5335 Support multiple keys in a PGP keyring > - NIFI-2983 Support decryption of signed messages > - NIFI-1694 Support keyring or individual key for encryption or decryption -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8251) Add Encrypt and Decrypt PGP Processors and Services
[ https://issues.apache.org/jira/browse/NIFI-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329385#comment-17329385 ] ASF subversion and git services commented on NIFI-8251: --- Commit 92bdc23adbe867c93f6e43c6665e84108d743fb2 in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=92bdc23 ] NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors - Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard implementations - NIFI-7396 EncryptContentPGP writes encryption metadata attributes - NIFI-6708 Controller Services support ElGamal Public and Private Keys - NIFI-5346 Controller Services support Keyring Files and ASCII Key properties - NIFI-5335 Controller Services support multiple public or private keys from keyrings - NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets regardless of signing - NIFI-1694 Controller Services support individual key files or keyrings NIFI-8251 Refactored Public Key ID Property to Public Key Search NIFI-8251 Corrected handling of multiple Encrypted Data packets in DecryptContentPGP - Added unit tests for encryption and decryption with both password-based and public key - Added PGP NAR dependencies to nifi-assembly Signed-off-by: Nathan Gough This closes #4842. > Add Encrypt and Decrypt PGP Processors and Services > --- > > Key: NIFI-8251 > URL: https://issues.apache.org/jira/browse/NIFI-8251 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.13.0 >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Labels: encryption, gpg, pgp, security > Time Spent: 2h 20m > Remaining Estimate: 0h > > The purpose of this issue is to create new Processors and Controller Services > to perform encrypt and decrypt functions according to the OpenPGP > specification defined in [RFC 4880|https://tools.ietf.org/html/rfc4880]. This > functionality is implemented in the `EncryptContent` Processor, but the > current approach is difficult to configure and troubleshoot in some cases. > The new Controller Services should encapsulate management of keys and > keyrings to improve performance and provide better error handling. New > Controller Services would also support building additional Processors for > signing and verifying PGP messages as requested in NIFI-7322. Separating > encrypt and decrypt content functions into distinct Processors will provide a > clearer distinction between the properties necessary for configuration. > The implementation of these Processors and Controller Services should address > the issues described in the following issues: > - NIFI-7396 Add encryption metadata attributes > - NIFI-6708 Support DSA and ElGamal keys > - NIFI-5346 Specify PGP key as property > - NIFI-5335 Support multiple keys in a PGP keyring > - NIFI-2983 Support decryption of signed messages > - NIFI-1694 Support keyring or individual key for encryption or decryption -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-2983) EncryptContent PGP mode can't handle signed encrypted files
[ https://issues.apache.org/jira/browse/NIFI-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329383#comment-17329383 ] ASF subversion and git services commented on NIFI-2983: --- Commit 92bdc23adbe867c93f6e43c6665e84108d743fb2 in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=92bdc23 ] NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors - Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard implementations - NIFI-7396 EncryptContentPGP writes encryption metadata attributes - NIFI-6708 Controller Services support ElGamal Public and Private Keys - NIFI-5346 Controller Services support Keyring Files and ASCII Key properties - NIFI-5335 Controller Services support multiple public or private keys from keyrings - NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets regardless of signing - NIFI-1694 Controller Services support individual key files or keyrings NIFI-8251 Refactored Public Key ID Property to Public Key Search NIFI-8251 Corrected handling of multiple Encrypted Data packets in DecryptContentPGP - Added unit tests for encryption and decryption with both password-based and public key - Added PGP NAR dependencies to nifi-assembly Signed-off-by: Nathan Gough This closes #4842. > EncryptContent PGP mode can't handle signed encrypted files > --- > > Key: NIFI-2983 > URL: https://issues.apache.org/jira/browse/NIFI-2983 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Environment: Apache NiFi 1.0.0 on Ubuntu 16.04 LTS. >Reporter: Emil Frank >Assignee: David Handermann >Priority: Minor > > The EncryptContent processor can't handle signed and encrypted PGP files - > just encrypted files work absolutely fine. > The error returned when an additionally signed file is (think it comes from > bouncycastle): > "encrypted message contains signed message - not literal data" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] thenatog closed pull request #4842: NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors
thenatog closed pull request #4842: URL: https://github.com/apache/nifi/pull/4842 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4842: NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors
thenatog commented on pull request #4842: URL: https://github.com/apache/nifi/pull/4842#issuecomment-825137621 Tested and looks good to me, +1 will merge. Thanks exceptionfactory, awesome work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-8465) Correct intermittent failures in PasswordBasedEncryptorGroovyTest
[
https://issues.apache.org/jira/browse/NIFI-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Grey reassigned NIFI-8465:
---
Assignee: Paul Grey
> Correct intermittent failures in PasswordBasedEncryptorGroovyTest
> -
>
> Key: NIFI-8465
> URL: https://issues.apache.org/jira/browse/NIFI-8465
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: David Handermann
>Assignee: Paul Grey
>Priority: Minor
> Labels: encryption, security
>
> {{PasswordBasedEncryptorGroovyTest}} can fail intermittently during automated
> builds. A recent failure occurred when comparing encrypted and plaintext
> results:
> {quote}Error: Failures:
> Error:
> PasswordBasedEncryptorGroovyTest.testBcryptDecryptShouldSupportLegacyKeyDerivationProcess:445
> assert recovered == PLAINTEXT{quote}
> The test class should be evaluated to determine the source of the problem and
> refactored to avoid potential failures. It is also worth evaluating the
> opportunity to refactor the test into more discrete test classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #1046: MINIFICPP-1032 Refactor parse_url
arpadboda commented on a change in pull request #1046:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1046#discussion_r618661909
##
File path: libminifi/src/utils/HTTPClient.cpp
##
@@ -15,8 +15,43 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include "utils/HTTPClient.h"
+#include
#include
+
+#include "utils/HTTPClient.h"
+#include "utils/StringUtils.h"
+
+namespace {
+
+constexpr const char* HTTP = "http://;;
+constexpr const char* HTTPS = "https://;;
+
+utils::optional parseProtocol(const std::string& url_input) {
+ if (utils::StringUtils::startsWith(url_input, HTTP)) {
+return HTTP;
+ } else if (utils::StringUtils::startsWith(url_input, HTTPS)) {
+return HTTPS;
+ } else {
+return {};
+ }
+}
+
+utils::optional parsePortNumber(const std::string& port_string) {
+ try {
+size_t pos;
+int port = std::stoi(port_string, );
+if (pos == port_string.size()) {
Review comment:
A safety trim before doing this?
"432 " would not be parsed as a port because of the ending space, but
debugging would be a nightmare.
##
File path: libminifi/include/utils/HTTPClient.h
##
@@ -362,10 +363,27 @@ class BaseHTTPClient {
virtual inline bool matches(const std::string , const std::string
) = 0;
};
-extern std::string get_token(utils::BaseHTTPClient *client, std::string
username, std::string password);
+std::string get_token(utils::BaseHTTPClient *client, std::string username,
std::string password);
+
+class URL {
Review comment:
I really like the strict typing for this instead of using string and
port numbers!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #1054: MINIFICPP-1544 Add Custom Text property to GenerateFlowFile processor
arpadboda commented on a change in pull request #1054:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1054#discussion_r618652706
##
File path: extensions/standard-processors/processors/GenerateFlowFile.cpp
##
@@ -61,6 +62,13 @@ core::Property GenerateFlowFile::UniqueFlowFiles(
core::PropertyBuilder::createProperty("Unique
FlowFiles")->withDescription("If true, each FlowFile that is generated will be
unique. If false, a random value will be generated and all FlowFiles")
->isRequired(false)->withDefaultValue(true)->build());
+core::Property GenerateFlowFile::CustomText(
+core::PropertyBuilder::createProperty("Custom Text")
+ ->withDescription("If Data Format is text and if Unique FlowFiles is
false, then this custom text will be used as content of the generated FlowFiles
and the File Size will be ignored. "
+"Finally, if Expression Language is used, evaluation
will be performed only once per batch of generated FlowFiles")
+ ->supportsExpressionLanguage(true)
Review comment:
+1
##
File path: extensions/standard-processors/processors/GenerateFlowFile.cpp
##
@@ -110,6 +119,15 @@ void GenerateFlowFile::onSchedule(const
std::shared_ptr
logger_->log_trace("Unique Flow files is configured to be %i",
uniqueFlowFile_);
}
+ if (textData_ && !uniqueFlowFile_) {
+std::string custom_text;
+context->getProperty(CustomText, custom_text, nullptr);
+if (!custom_text.empty()) {
+ std::copy(custom_text.begin(), custom_text.end(),
std::back_inserter(data_));
Review comment:
```
data_.assign(custom_text.begin(), custom_text.end());
```
For me this feels like a bit shorter and more efficient, but I may missed
some constraints here when using this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #1058: MINIFICPP-1547 - Change default c2 protocol
arpadboda commented on a change in pull request #1058: URL: https://github.com/apache/nifi-minifi-cpp/pull/1058#discussion_r618650912 ## File path: C2.md ## @@ -63,11 +63,11 @@ be requested via C2 DESCRIBE manifest command. # specify classes for the AST response nifi.c2.root.classes=DeviceInfoNode,AgentInformation,FlowInformation - # specify C2 protocol -- default is CoapProtocol if not specified - #nifi.c2.agent.protocol.class=CoapProtocol - # may also use MQTT or REST - # nifi.c2.agent.protocol.class=MQTTC2Protocol + # specify C2 protocol -- default is RESTSender if not specified nifi.c2.agent.protocol.class=RESTSender + # may also use MQTT or CoapProtocol + # nifi.c2.agent.protocol.class=MQTTC2Protocol Review comment: Do we have MQTT tested or at least documented anywhere? Honestly, I have never tried it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] granthenke commented on a change in pull request #5020: NIFI-8435 Added Kudu Client Worker Count property
granthenke commented on a change in pull request #5020:
URL: https://github.com/apache/nifi/pull/5020#discussion_r618639929
##
File path:
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKuduProcessor.java
##
@@ -184,10 +200,25 @@ protected KuduClient buildClient(final ProcessContext
context) {
final String masters =
context.getProperty(KUDU_MASTERS).evaluateAttributeExpressions().getValue();
final int operationTimeout =
context.getProperty(KUDU_OPERATION_TIMEOUT_MS).evaluateAttributeExpressions().asTimePeriod(TimeUnit.MILLISECONDS).intValue();
final int adminOperationTimeout =
context.getProperty(KUDU_KEEP_ALIVE_PERIOD_TIMEOUT_MS).evaluateAttributeExpressions().asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+final int workerCount = context.getProperty(WORKER_COUNT).asInteger();
+
+// Create Executor following approach of
Executors.newCachedThreadPool() using worker count as maximum pool size
+final int corePoolSize = 0;
+final long threadKeepAliveTime = 60;
+final Executor nioExecutor = new ThreadPoolExecutor(
Review comment:
That makes sense. Thanks for clarifying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8465) Correct intermittent failures in PasswordBasedEncryptorGroovyTest
David Handermann created NIFI-8465:
--
Summary: Correct intermittent failures in
PasswordBasedEncryptorGroovyTest
Key: NIFI-8465
URL: https://issues.apache.org/jira/browse/NIFI-8465
Project: Apache NiFi
Issue Type: Bug
Reporter: David Handermann
{{PasswordBasedEncryptorGroovyTest}} can fail intermittently during automated
builds. A recent failure occurred when comparing encrypted and plaintext
results:
{quote}Error: Failures:
Error:
PasswordBasedEncryptorGroovyTest.testBcryptDecryptShouldSupportLegacyKeyDerivationProcess:445
assert recovered == PLAINTEXT{quote}
The test class should be evaluated to determine the source of the problem and
refactored to avoid potential failures. It is also worth evaluating the
opportunity to refactor the test into more discrete test classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5020: NIFI-8435 Added Kudu Client Worker Count property
exceptionfactory commented on a change in pull request #5020:
URL: https://github.com/apache/nifi/pull/5020#discussion_r618628897
##
File path:
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKuduProcessor.java
##
@@ -184,10 +200,25 @@ protected KuduClient buildClient(final ProcessContext
context) {
final String masters =
context.getProperty(KUDU_MASTERS).evaluateAttributeExpressions().getValue();
final int operationTimeout =
context.getProperty(KUDU_OPERATION_TIMEOUT_MS).evaluateAttributeExpressions().asTimePeriod(TimeUnit.MILLISECONDS).intValue();
final int adminOperationTimeout =
context.getProperty(KUDU_KEEP_ALIVE_PERIOD_TIMEOUT_MS).evaluateAttributeExpressions().asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+final int workerCount = context.getProperty(WORKER_COUNT).asInteger();
+
+// Create Executor following approach of
Executors.newCachedThreadPool() using worker count as maximum pool size
+final int corePoolSize = 0;
+final long threadKeepAliveTime = 60;
+final Executor nioExecutor = new ThreadPoolExecutor(
+corePoolSize,
+workerCount,
+threadKeepAliveTime,
+TimeUnit.SECONDS,
+new SynchronousQueue<>(),
+new ClientThreadFactory(getIdentifier())
+);
return new KuduClient.KuduClientBuilder(masters)
.defaultOperationTimeoutMs(operationTimeout)
-.defaultSocketReadTimeoutMs(adminOperationTimeout)
+.defaultAdminOperationTimeoutMs(adminOperationTimeout)
Review comment:
Reverted the change for now, something that can be addressed separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5020: NIFI-8435 Added Kudu Client Worker Count property
exceptionfactory commented on a change in pull request #5020:
URL: https://github.com/apache/nifi/pull/5020#discussion_r618601572
##
File path:
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/PutKudu.java
##
@@ -342,38 +346,73 @@ public void onTrigger(final ProcessContext context, final
ProcessSession session
final KerberosUser user = getKerberosUser();
if (user == null) {
-executeOnKuduClient(kuduClient -> trigger(context, session,
flowFiles, kuduClient));
+executeOnKuduClient(kuduClient -> processFlowFiles(context,
session, flowFiles, kuduClient));
return;
}
final PrivilegedExceptionAction privilegedAction = () -> {
-executeOnKuduClient(kuduClient -> trigger(context, session,
flowFiles, kuduClient));
+executeOnKuduClient(kuduClient -> processFlowFiles(context,
session, flowFiles, kuduClient));
return null;
};
final KerberosAction action = new KerberosAction<>(user,
privilegedAction, getLogger());
action.execute();
}
-private void trigger(final ProcessContext context, final ProcessSession
session, final List flowFiles, KuduClient kuduClient) throws
ProcessException {
-final RecordReaderFactory recordReaderFactory =
context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+private void processFlowFiles(final ProcessContext context, final
ProcessSession session, final List flowFiles, final KuduClient
kuduClient) {
+final Map processedRecords = new HashMap<>();
+final Map flowFileFailures = new HashMap<>();
+final Map operationFlowFileMap = new HashMap<>();
+final List pendingRowErrors = new ArrayList<>();
final KuduSession kuduSession = createKuduSession(kuduClient);
+try {
+processFlowFiles(flowFiles,
+processedRecords,
+flowFileFailures,
+operationFlowFileMap,
+pendingRowErrors,
+session,
+context,
+kuduClient,
+kuduSession);
+} finally {
+try {
+flushKuduSession(kuduSession, true, pendingRowErrors);
+} catch (final KuduException|RuntimeException e) {
+getLogger().error("KuduSession.close() Failed", e);
+}
+}
-final Map numRecords = new HashMap<>();
-final Map flowFileFailures = new HashMap<>();
-final Map operationFlowFileMap = new HashMap<>();
+if (isRollbackOnFailure() && (!pendingRowErrors.isEmpty() ||
!flowFileFailures.isEmpty())) {
+logFailures(pendingRowErrors, operationFlowFileMap);
+session.rollback();
+context.yield();
+} else {
+transferFlowFiles(flowFiles, processedRecords, flowFileFailures,
operationFlowFileMap, pendingRowErrors, session);
+}
+}
-int numBuffered = 0;
+private void processFlowFiles(final List flowFiles,
Review comment:
Updated to processRecords, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (NIFI-8463) Custom SASL protocol name support in PutKudu processor
[ https://issues.apache.org/jira/browse/NIFI-8463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned NIFI-8463: -- Assignee: Attila Bukor > Custom SASL protocol name support in PutKudu processor > -- > > Key: NIFI-8463 > URL: https://issues.apache.org/jira/browse/NIFI-8463 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.14.0 >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > KUDU-1884 allows configuring the service principal name for the Kudu > services. The SASL protocol/service name must match the SPN base in the > client to be able to connect to the cluster. This needs to be configurable in > the PutKudu processor as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8460) assess need for Bouncycastle reference in unit test CatchAllFilterTest
[ https://issues.apache.org/jira/browse/NIFI-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329279#comment-17329279 ] ASF subversion and git services commented on NIFI-8460: --- Commit a1597ced890a5a1570ad7a7f9dcaac0d62dbbd9a in nifi's branch refs/heads/main from Paul Grey [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a1597ce ] NIFI-8460 Remove unused BouncyCastle reference in test class This closes #5022 Signed-off-by: David Handermann > assess need for Bouncycastle reference in unit test CatchAllFilterTest > -- > > Key: NIFI-8460 > URL: https://issues.apache.org/jira/browse/NIFI-8460 > Project: Apache NiFi > Issue Type: Task >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Trivial > Time Spent: 10m > Remaining Estimate: 0h > > [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-error/src/test/groovy/org/apache/nifi/web/filter/CatchAllFilterTest.groovy#L20] > > If provider is not needed for test, reference should be removed. If needed, > consider adding BC reference to project POM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8460) assess need for Bouncycastle reference in unit test CatchAllFilterTest
[ https://issues.apache.org/jira/browse/NIFI-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-8460. Fix Version/s: 1.14.0 Resolution: Fixed > assess need for Bouncycastle reference in unit test CatchAllFilterTest > -- > > Key: NIFI-8460 > URL: https://issues.apache.org/jira/browse/NIFI-8460 > Project: Apache NiFi > Issue Type: Task >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Trivial > Fix For: 1.14.0 > > Time Spent: 20m > Remaining Estimate: 0h > > [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-error/src/test/groovy/org/apache/nifi/web/filter/CatchAllFilterTest.groovy#L20] > > If provider is not needed for test, reference should be removed. If needed, > consider adding BC reference to project POM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #5022: NIFI-8460; remove unused BC reference
asfgit closed pull request #5022: URL: https://github.com/apache/nifi/pull/5022 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-3580) Setting SSL ciphers used by NiFi
[ https://issues.apache.org/jira/browse/NIFI-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-3580: --- Issue Type: Improvement (was: Bug) > Setting SSL ciphers used by NiFi > > > Key: NIFI-3580 > URL: https://issues.apache.org/jira/browse/NIFI-3580 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI > Environment: Linux >Reporter: Alexander M >Assignee: Paul Grey >Priority: Major > Labels: features, security > Time Spent: 3.5h > Remaining Estimate: 0h > > Does NiFi have an ability to choose which OpenSSL ciphers suites use in web > config (web UI https port / cluster interconnect port)? > I know that Jetty have this ability from the box, but I can't find related > parameters in NiFi configuration and documentation. > Can you turn me to the right way if this possible to change default ciphers > suite set or open a feature request for this possibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-3580) Setting SSL ciphers used by NiFi
[ https://issues.apache.org/jira/browse/NIFI-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-3580. Fix Version/s: 1.14.0 Resolution: Fixed > Setting SSL ciphers used by NiFi > > > Key: NIFI-3580 > URL: https://issues.apache.org/jira/browse/NIFI-3580 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI > Environment: Linux >Reporter: Alexander M >Assignee: Paul Grey >Priority: Major > Labels: features, security > Fix For: 1.14.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Does NiFi have an ability to choose which OpenSSL ciphers suites use in web > config (web UI https port / cluster interconnect port)? > I know that Jetty have this ability from the box, but I can't find related > parameters in NiFi configuration and documentation. > Can you turn me to the right way if this possible to change default ciphers > suite set or open a feature request for this possibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-3580) Setting SSL ciphers used by NiFi
[ https://issues.apache.org/jira/browse/NIFI-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329265#comment-17329265 ] ASF subversion and git services commented on NIFI-3580: --- Commit 17fa0cf3c1a509ad87386127def9373eb0c1e5df in nifi's branch refs/heads/main from Paul Grey [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=17fa0cf ] NIFI-3580 Add configurable TLS Cipher Suite properties This closes #5018 Signed-off-by: David Handermann > Setting SSL ciphers used by NiFi > > > Key: NIFI-3580 > URL: https://issues.apache.org/jira/browse/NIFI-3580 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI > Environment: Linux >Reporter: Alexander M >Assignee: Paul Grey >Priority: Major > Labels: features, security > Time Spent: 3h 20m > Remaining Estimate: 0h > > Does NiFi have an ability to choose which OpenSSL ciphers suites use in web > config (web UI https port / cluster interconnect port)? > I know that Jetty have this ability from the box, but I can't find related > parameters in NiFi configuration and documentation. > Can you turn me to the right way if this possible to change default ciphers > suite set or open a feature request for this possibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #5018: NIFI-3580 - configure TLS cipher suites
asfgit closed pull request #5018: URL: https://github.com/apache/nifi/pull/5018 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] attilabukor edited a comment on pull request #5023: NIFI-8463: Support custom SASL protocol name in PutKudu
attilabukor edited a comment on pull request #5023: URL: https://github.com/apache/nifi/pull/5023#issuecomment-824975337 We need to wait until Apache Kudu 1.15.0 is released as it depends on new functionality. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] attilabukor commented on pull request #5023: NIFI-8463: Support custom SASL protocol name in PutKudu
attilabukor commented on pull request #5023: URL: https://github.com/apache/nifi/pull/5023#issuecomment-824975337 We need to wait until apache/kudu 1.15.0 is released as it depends on new functionality. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] attilabukor opened a new pull request #5023: NIFI-8463: Support custom SASL protocol name in PutKudu
attilabukor opened a new pull request #5023:
URL: https://github.com/apache/nifi/pull/5023
Description of PR
As of KUDU-1884, Kudu supports custom Kerberos principals on server-side
and custom SASL protocol (service) names on client-side which must match
the SPN base, i.e. if the SPN is kudu/_HOST, SASL protocol name *must*
be "kudu" in the client to be able to connect to the cluster.
This patch adds the ability to configure this in the PutKudu processor.
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [x] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-8464) Custom SASL protocol name in PutKudu processor
[ https://issues.apache.org/jira/browse/NIFI-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved NIFI-8464. Resolution: Duplicate > Custom SASL protocol name in PutKudu processor > -- > > Key: NIFI-8464 > URL: https://issues.apache.org/jira/browse/NIFI-8464 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Attila Bukor >Priority: Major > > KUDU-1884 introduces custom Kerberos principals for Kudu servers and the > ability to configure the SASL protocol/service name on the client-side, which > must match the SPN base to be able to connect to the Kudu cluster. This must > be configurable in the NiFi PutKudu processor as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8464) Custom SASL protocol name in PutKudu processor
Attila Bukor created NIFI-8464: -- Summary: Custom SASL protocol name in PutKudu processor Key: NIFI-8464 URL: https://issues.apache.org/jira/browse/NIFI-8464 Project: Apache NiFi Issue Type: Improvement Reporter: Attila Bukor KUDU-1884 introduces custom Kerberos principals for Kudu servers and the ability to configure the SASL protocol/service name on the client-side, which must match the SPN base to be able to connect to the Kudu cluster. This must be configurable in the NiFi PutKudu processor as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8463) Custom SASL protocol name support in PutKudu processor
Attila Bukor created NIFI-8463: -- Summary: Custom SASL protocol name support in PutKudu processor Key: NIFI-8463 URL: https://issues.apache.org/jira/browse/NIFI-8463 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.14.0 Reporter: Attila Bukor KUDU-1884 allows configuring the service principal name for the Kudu services. The SASL protocol/service name must match the SPN base in the client to be able to connect to the cluster. This needs to be configurable in the PutKudu processor as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] greyp9 opened a new pull request #5022: NIFI-8460; remove unused BC reference
greyp9 opened a new pull request #5022:
URL: https://github.com/apache/nifi/pull/5022
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Remove unused reference to BouncyCastle library. This caused an issue when
importing POM into a fresh IntelliJ project.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-8459) Logout not working properly with OIDC when using Auth0 as provider
[ https://issues.apache.org/jira/browse/NIFI-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] R Arora updated NIFI-8459: -- Description: Hi, Not sure if this a bug... I have setup a OIDC configuration with Auth0 which has worked successfully. However, when I click on logout, I'm not redirected to the auth page. Instead I'm given a message "You have have successfully logged out. You may now close the window."... But when I click the home button or refresh the page, I get logged in again without going to the auth page to ask for user creds. >From what I'm reading in this blog: >[https://www.jerriepelser.com/blog/using-auth0-with-vue-oidc-client-js/] auth0 >doesn't provide end_session_endpoint in it's openid-configuration and hence >NiFi is confused on logout. Is there anything we can do to work around this? Any help is appreciated. Thanks! was: Hi, I have setup a OIDC configuration with Auth0 which has worked successfully. However, when I click on logout, I'm not redirected to the auth page. Instead I'm given a message "You have have successfully logged out. You may now close the window."... But when I click the home button or refresh the page, I get logged in again without going to the auth page to ask for user creds. >From what I'm reading in this blog: >https://www.jerriepelser.com/blog/using-auth0-with-vue-oidc-client-js/ auth0 >doesn't provide end_session_endpoint in it's openid-configuration and hence >NiFi is confused on logout. Is there anything we can do to work around this? Any help is appreciated. Thanks! > Logout not working properly with OIDC when using Auth0 as provider > -- > > Key: NIFI-8459 > URL: https://issues.apache.org/jira/browse/NIFI-8459 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.13.2 > Environment: Browser: Chrome / Firefox > Configuration of NiFi: > - SSL certificate for the server (no client auth) > - OIDC configuration using auth0 endpoint >Reporter: R Arora >Priority: Minor > > Hi, > Not sure if this a bug... > I have setup a OIDC configuration with Auth0 which has worked successfully. > However, when I click on logout, I'm not redirected to the auth page. Instead > I'm given a message "You have have successfully logged out. You may now close > the window."... But when I click the home button or refresh the page, I get > logged in again without going to the auth page to ask for user creds. > From what I'm reading in this blog: > [https://www.jerriepelser.com/blog/using-auth0-with-vue-oidc-client-js/] > auth0 doesn't provide end_session_endpoint in it's openid-configuration and > hence NiFi is confused on logout. Is there anything we can do to work around > this? > Any help is appreciated. > Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
lordgamez commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618508219
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -653,29 +688,57 @@ void TailFile::onTrigger(const
std::shared_ptr &, const st
if (!session->existsFlowFileInRelationship(Success)) {
yield();
}
+
+ first_trigger_ = false;
+}
+
+bool TailFile::isOldFileInitiallyRead(TailState ) const {
+ // This is our initial processing and no stored state was found
+ return first_trigger_ && state.last_read_time_ ==
std::chrono::system_clock::time_point{};
}
void TailFile::processFile(const std::shared_ptr
,
const std::string _file_name,
TailState ) {
- uint64_t fsize = utils::file::FileUtils::file_size(full_file_name);
- if (fsize < state.position_) {
-processRotatedFiles(session, state);
- } else if (fsize == state.position_) {
-logger_->log_trace("Skipping file %s as its size hasn't change since last
read", state.file_name_);
-return;
+ if (isOldFileInitiallyRead(state)) {
+if (initial_start_position_ == InitialStartPositions::BEGINNING_OF_TIME) {
+ processAllRotatedFiles(session, state);
+} else if (initial_start_position_ == InitialStartPositions::CURRENT_TIME)
{
+ state.position_ = utils::file::FileUtils::file_size(full_file_name);
+ state.last_read_time_ = std::chrono::system_clock::now();
+ state.checksum_ =
utils::file::FileUtils::computeChecksum(full_file_name, state.position_);
+ storeState();
+ return;
+}
+ } else {
+uint64_t fsize = utils::file::FileUtils::file_size(full_file_name);
+if (fsize < state.position_) {
+ processRotatedFilesAfterLastReadTime(session, state);
+} else if (fsize == state.position_) {
+ logger_->log_trace("Skipping file %s as its size hasn't change since
last read", state.file_name_);
+ return;
+}
}
processSingleFile(session, full_file_name, state);
}
-void TailFile::processRotatedFiles(const std::shared_ptr
, TailState ) {
-std::vector rotated_file_states = findRotatedFiles(state);
-for (TailState _state : rotated_file_states) {
- processSingleFile(session, file_state.fileNameWithPath(), file_state);
-}
-state.position_ = 0;
-state.checksum_ = 0;
+void TailFile::processRotatedFilesAfterLastReadTime(const
std::shared_ptr , TailState ) {
+ std::vector rotated_file_states =
findRotatedFilesAfterLastReadTime(state);
+ processRotatedFiles(session, state, rotated_file_states);
+}
+
+void TailFile::processAllRotatedFiles(const
std::shared_ptr , TailState ) {
+ std::vector rotated_file_states = findAllRotatedFiles(state);
+ processRotatedFiles(session, state, rotated_file_states);
+}
+
+void TailFile::processRotatedFiles(const std::shared_ptr
, TailState , std::vector _file_states) {
+ for (TailState _state : rotated_file_states) {
+processSingleFile(session, file_state.fileNameWithPath(), file_state);
Review comment:
It should be for a restart scenario, when the agent is restarted we
should know that those files are already processed and not new files, as in
that case we do not check for rollover patterns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
lordgamez commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618506643
##
File path: extensions/standard-processors/processors/TailFile.h
##
@@ -91,9 +99,12 @@ class TailFile : public core::Processor {
static core::Property RecursiveLookup;
static core::Property LookupFrequency;
static core::Property RollingFilenamePattern;
+ static core::Property InitialStartPosition;
// Supported Relationships
static core::Relationship Success;
+ static const std::set INITIAL_START_POSITIONS;
Review comment:
No, I left it there by accident when fixing the comment, thanks! Removed
in 393f540a9b21e7a3a8d8de78e3040519bac244af
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
lordgamez commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618505287
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -653,29 +690,56 @@ void TailFile::onTrigger(const
std::shared_ptr &, const st
if (!session->existsFlowFileInRelationship(Success)) {
yield();
}
+
+ first_trigger_ = false;
+}
+
+bool TailFile::isOldFileInitiallyRead(TailState ) const {
+ // This is our initial processing and no stored state was found
+ return first_trigger_ && state.checksum_ == 0;
}
void TailFile::processFile(const std::shared_ptr
,
const std::string _file_name,
TailState ) {
- uint64_t fsize = utils::file::FileUtils::file_size(full_file_name);
- if (fsize < state.position_) {
-processRotatedFiles(session, state);
- } else if (fsize == state.position_) {
-logger_->log_trace("Skipping file %s as its size hasn't change since last
read", state.file_name_);
-return;
+ if (isOldFileInitiallyRead(state)) {
+if (initial_start_position_ == "Beginning of Time") {
+ processAllRotatedFiles(session, state);
+} else if (initial_start_position_ == "Current Time") {
+ state.position_ = utils::file::FileUtils::file_size(full_file_name);
+ state.last_read_time_ = std::chrono::system_clock::now();
Review comment:
It seems that as I previously created a rolled over file for all tests
which was later rewritten in the rollover scenario with the same filename and
it was identified as a new file on Windows. Extracted the new test as a
separate test case in cbd9e2175f650749134ff7cf66dcd672aa3eb03c where I removed
this from the test setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8462) Refactor PutSyslog and ListenSyslog using Netty
David Handermann created NIFI-8462:
--
Summary: Refactor PutSyslog and ListenSyslog using Netty
Key: NIFI-8462
URL: https://issues.apache.org/jira/browse/NIFI-8462
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 1.13.2
Reporter: David Handermann
Assignee: David Handermann
Multiple extension and framework components leverage custom socket handling
classes for sending and receiving messages. These custom classes include
{{SSLSocketChannel}}, which handles handshaking and interaction with
{{SSLEngine}}. Custom {{SSLEngine}} handling is prone to errors, and the
current implementation has issues with TLS 1.3. Rather than continuing to
maintain custom {{SSLEngine}} handling, moving to a solution based on Netty
provides a cleaner approach.
The purpose of this issue is to refactor {{PutSyslog}} and {{ListenSyslog}}
using Netty to provide a way forward for other impacted components. The
changes should maintain compatibility with existing TCP and UDP protocol
configurations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] thenatog commented on a change in pull request #4988: NIFI-7870 - Fix anonymous access control for advanced UI resources
thenatog commented on a change in pull request #4988:
URL: https://github.com/apache/nifi/pull/4988#discussion_r618475268
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/AccessResource.java
##
@@ -1812,4 +1826,25 @@ public void setLogoutRequestManager(LogoutRequestManager
logoutRequestManager) {
this.logoutRequestManager = logoutRequestManager;
}
+private void logOutUser(HttpServletRequest httpServletRequest) throws
LogoutException {
+final String jwt = getJwtFromRequest(httpServletRequest);
+jwtService.logOut(jwt);
+}
+
+private String getJwtFromRequest(HttpServletRequest httpServletRequest) {
+final String authCookie =
getCookieValue(httpServletRequest.getCookies(),
JwtAuthenticationFilter.JWT_COOKIE_NAME);
+final String authHeader =
httpServletRequest.getHeader(httpServletRequest.getHeader(JwtAuthenticationFilter.AUTHORIZATION));
+
+String jwt = null;
+
+if (authCookie != null && !authCookie.isEmpty()) {
+jwt = authCookie;
+}
+
+if (authHeader != null && !authHeader.isEmpty()) {
+jwt = JwtAuthenticationFilter.getTokenFromHeader(authHeader);
Review comment:
Added a BearerTokenResolver/interface as this Spring class is not
introduced until Spring Security 5.1 and we are using 4.2.13.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-8461) Queue reports items but cannot list them
[ https://issues.apache.org/jira/browse/NIFI-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Aagaard updated NIFI-8461: Attachment: (was: nifi_flow_issue_1.jpg) > Queue reports items but cannot list them > > > Key: NIFI-8461 > URL: https://issues.apache.org/jira/browse/NIFI-8461 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 > Environment: Docker image apache/nifi :1.13.2, Windows 10 Pro, WSL 2 > (Ubuntu), x86_64 >Reporter: Kevin Aagaard >Priority: Major > Labels: queue > Attachments: image-2021-04-22-08-48-58-117.png, > image-2021-04-22-08-49-40-904.png > > > The following segment of workflow demonstrates the issue. The queue reports > that there are items within it, but they cannot be listed, even after > stopping the consumer (and producer). Since this is in a Docker Container, it > is unlikely an OS error. > > !image-2021-04-22-08-48-58-117.png! > > !image-2021-04-22-08-49-40-904.png! > I do not currently have a simplified example workflow to recreate the error, > but can work on it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8461) Queue reports items but cannot list them
[ https://issues.apache.org/jira/browse/NIFI-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Aagaard updated NIFI-8461: Attachment: (was: nifi_flow_issue_2.jpg) > Queue reports items but cannot list them > > > Key: NIFI-8461 > URL: https://issues.apache.org/jira/browse/NIFI-8461 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 > Environment: Docker image apache/nifi :1.13.2, Windows 10 Pro, WSL 2 > (Ubuntu), x86_64 >Reporter: Kevin Aagaard >Priority: Major > Labels: queue > Attachments: image-2021-04-22-08-48-58-117.png, > image-2021-04-22-08-49-40-904.png > > > The following segment of workflow demonstrates the issue. The queue reports > that there are items within it, but they cannot be listed, even after > stopping the consumer (and producer). Since this is in a Docker Container, it > is unlikely an OS error. > > !image-2021-04-22-08-48-58-117.png! > > !image-2021-04-22-08-49-40-904.png! > I do not currently have a simplified example workflow to recreate the error, > but can work on it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8459) Logout not working properly with OIDC when using Auth0 as provider
[
https://issues.apache.org/jira/browse/NIFI-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
R Arora updated NIFI-8459:
--
Description:
Hi,
I have setup a OIDC configuration with Auth0 which has worked successfully.
However, when I click on logout, I'm not redirected to the auth page. Instead
I'm given a message "You have have successfully logged out. You may now close
the window."... But when I click the home button or refresh the page, I get
logged in again without going to the auth page to ask for user creds.
>From what I'm reading in this blog:
>https://www.jerriepelser.com/blog/using-auth0-with-vue-oidc-client-js/ auth0
>doesn't provide end_session_endpoint in it's openid-configuration and hence
>NiFi is confused on logout. Is there anything we can do to work around this?
Any help is appreciated.
Thanks!
was:
Hi,
I have setup a OIDC configuration with Auth0 which has worked successfully.
However, when I click on logout, I'm not redirected to the auth page. Instead
I'm given a message "You have have successfully logged out. You may now close
the window."... But when I click the home button or refresh the page, I get
logged in again without going to the auth page to ask for user creds.
>From what I'm reading in this blog:
>[https://www.jerriepelser.com/blog/using-auth0-with-vue-oidc-client-js/] auth0
>doesn't provide {{end_session_endpoint in it's }}openid-configuration and
>hence NiFi is confused on logout. Is there anything we can do to work around
>this?
Any help is appreciated.
Thanks!
> Logout not working properly with OIDC when using Auth0 as provider
> --
>
> Key: NIFI-8459
> URL: https://issues.apache.org/jira/browse/NIFI-8459
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.13.2
> Environment: Browser: Chrome / Firefox
> Configuration of NiFi:
> - SSL certificate for the server (no client auth)
> - OIDC configuration using auth0 endpoint
>Reporter: R Arora
>Priority: Minor
>
> Hi,
>
> I have setup a OIDC configuration with Auth0 which has worked successfully.
> However, when I click on logout, I'm not redirected to the auth page. Instead
> I'm given a message "You have have successfully logged out. You may now close
> the window."... But when I click the home button or refresh the page, I get
> logged in again without going to the auth page to ask for user creds.
> From what I'm reading in this blog:
> https://www.jerriepelser.com/blog/using-auth0-with-vue-oidc-client-js/ auth0
> doesn't provide end_session_endpoint in it's openid-configuration and hence
> NiFi is confused on logout. Is there anything we can do to work around this?
> Any help is appreciated.
> Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-8460) assess need for Bouncycastle reference in unit test CatchAllFilterTest
Paul Grey created NIFI-8460: --- Summary: assess need for Bouncycastle reference in unit test CatchAllFilterTest Key: NIFI-8460 URL: https://issues.apache.org/jira/browse/NIFI-8460 Project: Apache NiFi Issue Type: Task Reporter: Paul Grey Assignee: Paul Grey [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-error/src/test/groovy/org/apache/nifi/web/filter/CatchAllFilterTest.groovy#L20] If provider is not needed for test, reference should be removed. If needed, consider adding BC reference to project POM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-8460) assess need for Bouncycastle reference in unit test CatchAllFilterTest
[ https://issues.apache.org/jira/browse/NIFI-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329168#comment-17329168 ] Paul Grey edited comment on NIFI-8460 at 4/22/21, 2:49 PM: --- Discovered this issue while importing NiFi POM into fresh IntelliJ project. was (Author: pgrey): Discovered this issue while importing NiFI POM into fresh IntelliJ project. > assess need for Bouncycastle reference in unit test CatchAllFilterTest > -- > > Key: NIFI-8460 > URL: https://issues.apache.org/jira/browse/NIFI-8460 > Project: Apache NiFi > Issue Type: Task >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Trivial > > [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-error/src/test/groovy/org/apache/nifi/web/filter/CatchAllFilterTest.groovy#L20] > > If provider is not needed for test, reference should be removed. If needed, > consider adding BC reference to project POM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8461) Queue reports items but cannot list them
[ https://issues.apache.org/jira/browse/NIFI-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Aagaard updated NIFI-8461: Attachment: image-2021-04-22-08-48-58-117.png image-2021-04-22-08-49-40-904.png Description: The following segment of workflow demonstrates the issue. The queue reports that there are items within it, but they cannot be listed, even after stopping the consumer (and producer). Since this is in a Docker Container, it is unlikely an OS error. !image-2021-04-22-08-48-58-117.png! !image-2021-04-22-08-49-40-904.png! I do not currently have a simplified example workflow to recreate the error, but can work on it. was: The following segment of workflow demonstrates the issue. The queue reports that there are items within it, but they cannot be listed, even after stopping the consumer (and producer). Since this is in a Docker Container, it is unlikely an OS error. I do not currently have a simplified example workflow to recreate the error, but can work on it. > Queue reports items but cannot list them > > > Key: NIFI-8461 > URL: https://issues.apache.org/jira/browse/NIFI-8461 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 > Environment: Docker image apache/nifi :1.13.2, Windows 10 Pro, WSL 2 > (Ubuntu), x86_64 >Reporter: Kevin Aagaard >Priority: Major > Labels: queue > Attachments: image-2021-04-22-08-48-58-117.png, > image-2021-04-22-08-49-40-904.png, nifi_flow_issue_1.jpg, > nifi_flow_issue_2.jpg > > > The following segment of workflow demonstrates the issue. The queue reports > that there are items within it, but they cannot be listed, even after > stopping the consumer (and producer). Since this is in a Docker Container, it > is unlikely an OS error. > > !image-2021-04-22-08-48-58-117.png! > > !image-2021-04-22-08-49-40-904.png! > I do not currently have a simplified example workflow to recreate the error, > but can work on it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8459) Logout not working properly with OIDC when using Auth0 as provider
[
https://issues.apache.org/jira/browse/NIFI-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
R Arora updated NIFI-8459:
--
Description:
Hi,
I have setup a OIDC configuration with Auth0 which has worked successfully.
However, when I click on logout, I'm not redirected to the auth page. Instead
I'm given a message "You have have successfully logged out. You may now close
the window."... But when I click the home button or refresh the page, I get
logged in again without going to the auth page to ask for user creds.
>From what I'm reading in this blog:
>[https://www.jerriepelser.com/blog/using-auth0-with-vue-oidc-client-js/] auth0
>doesn't provide {{end_session_endpoint in it's }}openid-configuration and
>hence NiFi is confused on logout. Is there anything we can do to work around
>this?
Any help is appreciated.
Thanks!
was:
Hi,
I have setup a OIDC configuration with Auth0 which has worked successfully.
However, when I click on logout, I'm not redirected to the auth page. Instead
I'm given a message "You have have successfully logged out. You may now close
the window."... But when I click the home button or refresh the page, I get
logged in again without going to the auth page to ask for user creds.
I'm not quite sure if this is a bug or how to work around this issue and get
NiFi to behave in a way that it would redirect to the auth page on logout and
re-ask for credentials.
Any help is appreciated.
Thanks!
> Logout not working properly with OIDC when using Auth0 as provider
> --
>
> Key: NIFI-8459
> URL: https://issues.apache.org/jira/browse/NIFI-8459
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.13.2
> Environment: Browser: Chrome / Firefox
> Configuration of NiFi:
> - SSL certificate for the server (no client auth)
> - OIDC configuration using auth0 endpoint
>Reporter: R Arora
>Priority: Minor
>
> Hi,
> I have setup a OIDC configuration with Auth0 which has worked successfully.
> However, when I click on logout, I'm not redirected to the auth page. Instead
> I'm given a message "You have have successfully logged out. You may now close
> the window."... But when I click the home button or refresh the page, I get
> logged in again without going to the auth page to ask for user creds.
> From what I'm reading in this blog:
> [https://www.jerriepelser.com/blog/using-auth0-with-vue-oidc-client-js/]
> auth0 doesn't provide {{end_session_endpoint in it's }}openid-configuration
> and hence NiFi is confused on logout. Is there anything we can do to work
> around this?
>
> Any help is appreciated.
> Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-8461) Queue reports items but cannot list them
Kevin Aagaard created NIFI-8461: --- Summary: Queue reports items but cannot list them Key: NIFI-8461 URL: https://issues.apache.org/jira/browse/NIFI-8461 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.13.2 Environment: Docker image apache/nifi :1.13.2, Windows 10 Pro, WSL 2 (Ubuntu), x86_64 Reporter: Kevin Aagaard Attachments: nifi_flow_issue_1.jpg, nifi_flow_issue_2.jpg The following segment of workflow demonstrates the issue. The queue reports that there are items within it, but they cannot be listed, even after stopping the consumer (and producer). Since this is in a Docker Container, it is unlikely an OS error. I do not currently have a simplified example workflow to recreate the error, but can work on it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8460) assess need for Bouncycastle reference in unit test CatchAllFilterTest
[ https://issues.apache.org/jira/browse/NIFI-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329168#comment-17329168 ] Paul Grey commented on NIFI-8460: - Discovered this issue while importing NiFI POM into fresh IntelliJ project. > assess need for Bouncycastle reference in unit test CatchAllFilterTest > -- > > Key: NIFI-8460 > URL: https://issues.apache.org/jira/browse/NIFI-8460 > Project: Apache NiFi > Issue Type: Task >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Trivial > > [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-error/src/test/groovy/org/apache/nifi/web/filter/CatchAllFilterTest.groovy#L20] > > If provider is not needed for test, reference should be removed. If needed, > consider adding BC reference to project POM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] tpalfy opened a new pull request #5021: NIFI-8458 TailFile - Fix some bugs. Add more tests.
tpalfy opened a new pull request #5021: URL: https://github.com/apache/nifi/pull/5021 https://issues.apache.org/jira/browse/NIFI-8458 ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with **NIFI-** where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically `main`)? - [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._ ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] Have you verified that the full build is successful on JDK 8? - [ ] Have you verified that the full build is successful on JDK 11? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`? - [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`? - [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-8459) Logout not working properly with OIDC when using Auth0 as provider
R Arora created NIFI-8459: - Summary: Logout not working properly with OIDC when using Auth0 as provider Key: NIFI-8459 URL: https://issues.apache.org/jira/browse/NIFI-8459 Project: Apache NiFi Issue Type: Bug Components: Core UI Affects Versions: 1.13.2 Environment: Browser: Chrome / Firefox Configuration of NiFi: - SSL certificate for the server (no client auth) - OIDC configuration using auth0 endpoint Reporter: R Arora Hi, I have setup a OIDC configuration with Auth0 which has worked successfully. However, when I click on logout, I'm not redirected to the auth page. Instead I'm given a message "You have have successfully logged out. You may now close the window."... But when I click the home button or refresh the page, I get logged in again without going to the auth page to ask for user creds. I'm not quite sure if this is a bug or how to work around this issue and get NiFi to behave in a way that it would redirect to the auth page on logout and re-ask for credentials. Any help is appreciated. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
adamdebreceni commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618457506
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -653,29 +688,57 @@ void TailFile::onTrigger(const
std::shared_ptr &, const st
if (!session->existsFlowFileInRelationship(Success)) {
yield();
}
+
+ first_trigger_ = false;
+}
+
+bool TailFile::isOldFileInitiallyRead(TailState ) const {
+ // This is our initial processing and no stored state was found
+ return first_trigger_ && state.last_read_time_ ==
std::chrono::system_clock::time_point{};
}
void TailFile::processFile(const std::shared_ptr
,
const std::string _file_name,
TailState ) {
- uint64_t fsize = utils::file::FileUtils::file_size(full_file_name);
- if (fsize < state.position_) {
-processRotatedFiles(session, state);
- } else if (fsize == state.position_) {
-logger_->log_trace("Skipping file %s as its size hasn't change since last
read", state.file_name_);
-return;
+ if (isOldFileInitiallyRead(state)) {
+if (initial_start_position_ == InitialStartPositions::BEGINNING_OF_TIME) {
+ processAllRotatedFiles(session, state);
+} else if (initial_start_position_ == InitialStartPositions::CURRENT_TIME)
{
+ state.position_ = utils::file::FileUtils::file_size(full_file_name);
+ state.last_read_time_ = std::chrono::system_clock::now();
+ state.checksum_ =
utils::file::FileUtils::computeChecksum(full_file_name, state.position_);
+ storeState();
+ return;
+}
+ } else {
+uint64_t fsize = utils::file::FileUtils::file_size(full_file_name);
+if (fsize < state.position_) {
+ processRotatedFilesAfterLastReadTime(session, state);
+} else if (fsize == state.position_) {
+ logger_->log_trace("Skipping file %s as its size hasn't change since
last read", state.file_name_);
+ return;
+}
}
processSingleFile(session, full_file_name, state);
}
-void TailFile::processRotatedFiles(const std::shared_ptr
, TailState ) {
-std::vector rotated_file_states = findRotatedFiles(state);
-for (TailState _state : rotated_file_states) {
- processSingleFile(session, file_state.fileNameWithPath(), file_state);
-}
-state.position_ = 0;
-state.checksum_ = 0;
+void TailFile::processRotatedFilesAfterLastReadTime(const
std::shared_ptr , TailState ) {
+ std::vector rotated_file_states =
findRotatedFilesAfterLastReadTime(state);
+ processRotatedFiles(session, state, rotated_file_states);
+}
+
+void TailFile::processAllRotatedFiles(const
std::shared_ptr , TailState ) {
+ std::vector rotated_file_states = findAllRotatedFiles(state);
+ processRotatedFiles(session, state, rotated_file_states);
+}
+
+void TailFile::processRotatedFiles(const std::shared_ptr
, TailState , std::vector _file_states) {
+ for (TailState _state : rotated_file_states) {
+processSingleFile(session, file_state.fileNameWithPath(), file_state);
Review comment:
`processSingleFile` calls `storeState`, is that desirable with rotated
files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (NIFI-7468) Improve internal handling of SSL channels
[
https://issues.apache.org/jira/browse/NIFI-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-7468:
---
Status: In Progress (was: Patch Available)
> Improve internal handling of SSL channels
> -
>
> Key: NIFI-7468
> URL: https://issues.apache.org/jira/browse/NIFI-7468
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Extensions
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Assignee: David Handermann
>Priority: Major
> Labels: security, ssl, tcp, tls, tlsv1.3, udp
> Time Spent: 40m
> Remaining Estimate: 0h
>
> While refactoring the TLS protocol version issue in NIFI-7407, I discovered
> that some processors make use of NiFi custom implementations of
> {{SSLSocketChannel}}, {{SSLCommsSession}}, and
> {{SSLSocketChannelInputStream}}. These implementations break on TLSv1.3.
> Further investigation is needed to determine why these custom implementations
> were provided originally, whether they are still required, and why they do
> not handle TLSv1.3 successfully.
> Diagnostic error:
> {code}
> Error reading from channel due to Tag mismatch!: javax.net.ssl.SSLException:
> Tag mismatch!
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] thenatog commented on a change in pull request #4988: NIFI-7870 - Fix anonymous access control for advanced UI resources
thenatog commented on a change in pull request #4988:
URL: https://github.com/apache/nifi/pull/4988#discussion_r618448529
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/jwt/JwtAuthenticationFilter.java
##
@@ -43,33 +51,63 @@ public Authentication attemptAuthentication(final
HttpServletRequest request) {
return null;
}
-// TODO: Refactor request header extraction logic to shared utility as
it is duplicated in AccessResource
+// Check for JWT in cookie and header
+final String cookieToken = getTokenFromCookie(request);
+final String headerToken = getTokenFromHeader(request);
+if (cookieToken != null && !cookieToken.isEmpty()) {
+if
(!UNAUTHENTICATED_METHODS.contains(request.getMethod().toUpperCase())) {
+// To protect against CSRF when using a cookie, if the request
method requires authentication the request must have a matching Authorization
header JWT
+if (headerToken.equals(cookieToken)) {
+return new JwtAuthenticationRequestToken(headerToken,
request.getRemoteAddr());
+} else {
+throw new InvalidAuthenticationException("Authorization
HTTP header and authentication cookie did not match.");
+}
+} else {
+return new JwtAuthenticationRequestToken(cookieToken,
request.getRemoteAddr());
+}
+} else if (headerToken != null && !headerToken.isEmpty()) {
Review comment:
I have refactored this by adding the NiFiCsrfTokenRepository which will
check the Cookie header for a 'CSRF' token that should match the JWT
Authorization header. This simplifies the JwtAuthenticationFilter and pushed
the idempotent methods checks to the Spring CsrfFilter code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits closed pull request #1029: MINIFICPP-1345 Add flake8 check for python files
fgerlits closed pull request #1029: URL: https://github.com/apache/nifi-minifi-cpp/pull/1029 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
lordgamez commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618425021
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -653,29 +690,56 @@ void TailFile::onTrigger(const
std::shared_ptr &, const st
if (!session->existsFlowFileInRelationship(Success)) {
yield();
}
+
+ first_trigger_ = false;
+}
+
+bool TailFile::isOldFileInitiallyRead(TailState ) const {
+ // This is our initial processing and no stored state was found
+ return first_trigger_ && state.checksum_ == 0;
}
void TailFile::processFile(const std::shared_ptr
,
const std::string _file_name,
TailState ) {
- uint64_t fsize = utils::file::FileUtils::file_size(full_file_name);
- if (fsize < state.position_) {
-processRotatedFiles(session, state);
- } else if (fsize == state.position_) {
-logger_->log_trace("Skipping file %s as its size hasn't change since last
read", state.file_name_);
-return;
+ if (isOldFileInitiallyRead(state)) {
+if (initial_start_position_ == "Beginning of Time") {
+ processAllRotatedFiles(session, state);
+} else if (initial_start_position_ == "Current Time") {
+ state.position_ = utils::file::FileUtils::file_size(full_file_name);
+ state.last_read_time_ = std::chrono::system_clock::now();
Review comment:
For some reason on Windows the test case fails, have to investigate it
further
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
adamdebreceni commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618437327
##
File path: extensions/standard-processors/processors/TailFile.h
##
@@ -91,9 +99,12 @@ class TailFile : public core::Processor {
static core::Property RecursiveLookup;
static core::Property LookupFrequency;
static core::Property RollingFilenamePattern;
+ static core::Property InitialStartPosition;
// Supported Relationships
static core::Relationship Success;
+ static const std::set INITIAL_START_POSITIONS;
Review comment:
is this used somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-8423) Timezone wrong in UI for an 8 node cluster
[
https://issues.apache.org/jira/browse/NIFI-8423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt resolved NIFI-8423.
Resolution: Information Provided
The problem is being caused by scripted processors/custom processors which are
changing the system wide timezone/date settings. That will and has in the past
caused these types of problems. There are always other ways of manipulating
these timestamps/dates/etc.. as needed in the data without changing the system
wide settings.
> Timezone wrong in UI for an 8 node cluster
> --
>
> Key: NIFI-8423
> URL: https://issues.apache.org/jira/browse/NIFI-8423
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.13.2
> Environment: 8 Node NiFi Cluster on CentOS 7
> OpenJDK 1.8.0_282
> Local timezone: Europe/Zurich (CEST or UTC+2h)
>Reporter: Josef Zahner
>Priority: Critical
> Labels: centos, cluster, openjdk, timezone
> Attachments: Screenshot 2021-04-20 at 14.24.17.png, Screenshot
> 2021-04-20 at 14.34.36.png, Screenshot 2021-04-20 at 15.00.22.png,
> image-2021-04-13-15-14-02-162.png, image-2021-04-13-15-14-06-930.png,
> image-2021-04-13-15-14-56-690.png, manual_configured_timezone_gui_output.png,
> nifi-app_log.png
>
>
> We just upgraded to NiFi 1.13.2 and Java 1.8.0_282
> On our 8 node NiFi 1.13.2 cluster with timezone Europe/Zurich (CEST/CET), we
> have the issue that the UI does display the correct timezone (CEST, so UTC
> +2h), but in fact the time is displayed as UTC. NTP is enabled and working.
> The OS configuration/location is everywhere the same (doesn't matter if
> single or cluster NiFi). My tests below are all done at around 15:xx:xx local
> time (CEST).
> As you can see below, the timezone seems to be correct, but the time itself
> within NiFi is 2h behind (so in fact UTC) compared to Windows. In earlier
> NiFi/java versions it was enough to multiple times restart the cluster, but
> on the newest versions this doesn't help anymore. It shows most of the time
> CEST with the wrong time or directly UTC.
> !image-2021-04-13-15-14-06-930.png!
>
> The single NiFi instances or the 2 node clusters are always fine. The issue
> exists only on our 8 node cluster.
> NiFi Single Node Screenshot, which is fine (CEST, so UTC + 2h):
> !image-2021-04-13-15-14-02-162.png!
>
> If we set the -Duser.timezone to "Europe/Zurich" in bootstrap.conf, the UI
> shows no summer time, so only GMT+1 instead of GMT+2. As well not what we
> want.
> {code:java}
> java.arg.20=-Duser.timezone="Europe/Zurich"{code}
> !image-2021-04-13-15-14-56-690.png!
>
> What Matt below suggested has been verified, all servers (single nodes as
> well as clusters) are reporting the same time/timezone.
> [https://community.cloudera.com/t5/Support-Questions/NiFi-clock-is-off-by-one-hour-daylight-savings-problem/td-p/192942]
>
> So the question remains, where on a NiFi cluster comes the time from the UI
> and what could cause it that it is wrong? Sometimes I get UTC, sometimes I'm
> getting CEST but the time is anyhow UTC instead of CEST... I really need to
> have the correct time in the UI as I don't know what the impact could be on
> our dataflows.
>
> Any help would be really appreciated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7516) Predictions model throws intermittent SingularMatrixExceptions
[ https://issues.apache.org/jira/browse/NIFI-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-7516: --- Fix Version/s: 1.12.0 > Predictions model throws intermittent SingularMatrixExceptions > -- > > Key: NIFI-7516 > URL: https://issues.apache.org/jira/browse/NIFI-7516 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.12.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Under some circumstances, the Connection Status Analytics model (specifically > the Ordinary Least Squares model) throws a SingularMatrix exception: > org.apache.commons.math3.linear.SingularMatrixException: matrix is singular > This can happen (usually intermittently) when the data points used to update > the model form a matrix that has no inverse (i.e. singular). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFIREG-179) Move Flow to other buckets and rename them
[ https://issues.apache.org/jira/browse/NIFIREG-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17329136#comment-17329136 ] John Wise commented on NIFIREG-179: --- Ideally, the current flow UUID would be maintained, so that process groups deployed in a NiFi flow don't need to be replaced. From the command-line, nfcli's "registry sync-flow-versions" requires that a new destination UUID be created in the bucket to move an existing flow to, which is what creates the disconnect. Ideally, that command would just move the existing UUID from the old bucket to the new bucket. Any UI implementation would work the same way. > Move Flow to other buckets and rename them > -- > > Key: NIFIREG-179 > URL: https://issues.apache.org/jira/browse/NIFIREG-179 > Project: NiFi Registry > Issue Type: Improvement >Affects Versions: 0.1.0, 0.2.0 >Reporter: Julian Gimbel >Priority: Minor > > It should be possible to move a versioned flow from one bucket to another in > the UI. > In addition it should be possible to change the Flow Name for versioned flows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request #1059: MINIFICPP-1510 Register and fix InvokeHTTPTests
lordgamez opened a new pull request #1059: URL: https://github.com/apache/nifi-minifi-cpp/pull/1059 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1055: MINIFICPP-1454 Reduce duplication of CMake parameters in docker build
lordgamez commented on a change in pull request #1055:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1055#discussion_r618363196
##
File path: cmake/DockerConfig.cmake
##
@@ -22,7 +22,6 @@ add_custom_target(
-u 1000
-g 1000
-v
${PROJECT_VERSION_MAJOR}.${PROJECT_VERSION_MINOR}.${PROJECT_VERSION_PATCH}
--c IMAGE_TYPE=release
Review comment:
I think you are right, it should be the same parameter list everywhere
for the sake of consistency, I added the `-i` switch here as well in c802edc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8458) TailFile fails in certain scenarios
Tamas Palfy created NIFI-8458: - Summary: TailFile fails in certain scenarios Key: NIFI-8458 URL: https://issues.apache.org/jira/browse/NIFI-8458 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy Certain use cases can lead data loss or data duplication which need to be handled. Current tests are insufficient, we need to add more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5020: NIFI-8435 Added Kudu Client Worker Count property
exceptionfactory commented on a change in pull request #5020:
URL: https://github.com/apache/nifi/pull/5020#discussion_r618375958
##
File path:
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKuduProcessor.java
##
@@ -184,10 +200,25 @@ protected KuduClient buildClient(final ProcessContext
context) {
final String masters =
context.getProperty(KUDU_MASTERS).evaluateAttributeExpressions().getValue();
final int operationTimeout =
context.getProperty(KUDU_OPERATION_TIMEOUT_MS).evaluateAttributeExpressions().asTimePeriod(TimeUnit.MILLISECONDS).intValue();
final int adminOperationTimeout =
context.getProperty(KUDU_KEEP_ALIVE_PERIOD_TIMEOUT_MS).evaluateAttributeExpressions().asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+final int workerCount = context.getProperty(WORKER_COUNT).asInteger();
+
+// Create Executor following approach of
Executors.newCachedThreadPool() using worker count as maximum pool size
+final int corePoolSize = 0;
+final long threadKeepAliveTime = 60;
+final Executor nioExecutor = new ThreadPoolExecutor(
+corePoolSize,
+workerCount,
+threadKeepAliveTime,
+TimeUnit.SECONDS,
+new SynchronousQueue<>(),
+new ClientThreadFactory(getIdentifier())
+);
return new KuduClient.KuduClientBuilder(masters)
.defaultOperationTimeoutMs(operationTimeout)
-.defaultSocketReadTimeoutMs(adminOperationTimeout)
+.defaultAdminOperationTimeoutMs(adminOperationTimeout)
Review comment:
The `defaultSocketReadTimeoutMs()` setting is deprecated in version
1.14, and the existing variable name `adminOperationTimeout` seemed to be a
good match for the `defaultAdminOperationTimeoutMs()` setting. However, the
processor property name and description do not seem to match up as described.
Do you think `defaultAdminOperationTimeoutMs()` should be set using the
`operationTimeout` value instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
lordgamez commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618406371
##
File path: libminifi/include/utils/ProcessorConfigUtils.h
##
@@ -19,59 +19,23 @@
#include
#include
+#include
-#include "utils/StringUtils.h"
+#include "core/ProcessContext.h"
namespace org {
namespace apache {
namespace nifi {
namespace minifi {
namespace utils {
-std::string getRequiredPropertyOrThrow(const core::ProcessContext* context,
const std::string& property_name) {
- std::string value;
- if (!context->getProperty(property_name, value)) {
-throw std::runtime_error(property_name + " property missing or invalid");
- }
- return value;
-}
-
-std::vector listFromCommaSeparatedProperty(const
core::ProcessContext* context, const std::string& property_name) {
- std::string property_string;
- context->getProperty(property_name, property_string);
- return utils::StringUtils::splitAndTrim(property_string, ",");
-}
-
-std::vector listFromRequiredCommaSeparatedProperty(const
core::ProcessContext* context, const std::string& property_name) {
- return utils::StringUtils::splitAndTrim(getRequiredPropertyOrThrow(context,
property_name), ",");
-}
-
-bool parseBooleanPropertyOrThrow(core::ProcessContext* context, const
std::string& property_name) {
- const std::string value_str = getRequiredPropertyOrThrow(context,
property_name);
- utils::optional maybe_value = utils::StringUtils::toBool(value_str);
- if (!maybe_value) {
-throw std::runtime_error(property_name + " property is invalid: value is "
+ value_str);
- }
- return maybe_value.value();
-}
-
-std::chrono::milliseconds parseTimePropertyMSOrThrow(core::ProcessContext*
context, const std::string& property_name) {
- core::TimeUnit unit;
- uint64_t time_value_ms;
- const std::string value_str = getRequiredPropertyOrThrow(context,
property_name);
- if (!core::Property::StringToTime(value_str, time_value_ms, unit) ||
!core::Property::ConvertTimeUnitToMS(time_value_ms, unit, time_value_ms)) {
-throw std::runtime_error(property_name + " property is invalid: value is "
+ value_str);
- }
- return std::chrono::milliseconds(time_value_ms);
-}
-
-utils::optional getOptionalUintProperty(const core::ProcessContext&
context, const std::string& property_name) {
- uint64_t value;
- if (context.getProperty(property_name, value)) {
-return { value };
- }
- return utils::nullopt;
-}
+std::string getRequiredPropertyOrThrow(const core::ProcessContext* context,
const std::string& property_name);
Review comment:
It looks like it, maybe @hunyadi-dev can comment on it as it was
introduced in #940. Are there any differences in the behavior of getProperty
and getRequiredPropertyOrThrow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
lordgamez commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618360710
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -653,29 +690,56 @@ void TailFile::onTrigger(const
std::shared_ptr &, const st
if (!session->existsFlowFileInRelationship(Success)) {
yield();
}
+
+ first_trigger_ = false;
+}
+
+bool TailFile::isOldFileInitiallyRead(TailState ) const {
+ // This is our initial processing and no stored state was found
+ return first_trigger_ && state.checksum_ == 0;
}
void TailFile::processFile(const std::shared_ptr
,
const std::string _file_name,
TailState ) {
- uint64_t fsize = utils::file::FileUtils::file_size(full_file_name);
- if (fsize < state.position_) {
-processRotatedFiles(session, state);
- } else if (fsize == state.position_) {
-logger_->log_trace("Skipping file %s as its size hasn't change since last
read", state.file_name_);
-return;
+ if (isOldFileInitiallyRead(state)) {
+if (initial_start_position_ == "Beginning of Time") {
+ processAllRotatedFiles(session, state);
+} else if (initial_start_position_ == "Current Time") {
+ state.position_ = utils::file::FileUtils::file_size(full_file_name);
+ state.last_read_time_ = std::chrono::system_clock::now();
Review comment:
Thanks for the tip, I totally missed this, updated in
7a926a3372b44cfb32beb10e59730619a9154dbb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (NIFI-8458) TailFile fails in certain scenarios
[ https://issues.apache.org/jira/browse/NIFI-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy reassigned NIFI-8458: - Assignee: Tamas Palfy > TailFile fails in certain scenarios > --- > > Key: NIFI-8458 > URL: https://issues.apache.org/jira/browse/NIFI-8458 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Assignee: Tamas Palfy >Priority: Major > > Certain use cases can lead data loss or data duplication which need to be > handled. > Current tests are insufficient, we need to add more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
adamdebreceni commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618393554
##
File path: libminifi/include/utils/ProcessorConfigUtils.h
##
@@ -19,59 +19,23 @@
#include
#include
+#include
-#include "utils/StringUtils.h"
+#include "core/ProcessContext.h"
namespace org {
namespace apache {
namespace nifi {
namespace minifi {
namespace utils {
-std::string getRequiredPropertyOrThrow(const core::ProcessContext* context,
const std::string& property_name) {
- std::string value;
- if (!context->getProperty(property_name, value)) {
-throw std::runtime_error(property_name + " property missing or invalid");
- }
- return value;
-}
-
-std::vector listFromCommaSeparatedProperty(const
core::ProcessContext* context, const std::string& property_name) {
- std::string property_string;
- context->getProperty(property_name, property_string);
- return utils::StringUtils::splitAndTrim(property_string, ",");
-}
-
-std::vector listFromRequiredCommaSeparatedProperty(const
core::ProcessContext* context, const std::string& property_name) {
- return utils::StringUtils::splitAndTrim(getRequiredPropertyOrThrow(context,
property_name), ",");
-}
-
-bool parseBooleanPropertyOrThrow(core::ProcessContext* context, const
std::string& property_name) {
- const std::string value_str = getRequiredPropertyOrThrow(context,
property_name);
- utils::optional maybe_value = utils::StringUtils::toBool(value_str);
- if (!maybe_value) {
-throw std::runtime_error(property_name + " property is invalid: value is "
+ value_str);
- }
- return maybe_value.value();
-}
-
-std::chrono::milliseconds parseTimePropertyMSOrThrow(core::ProcessContext*
context, const std::string& property_name) {
- core::TimeUnit unit;
- uint64_t time_value_ms;
- const std::string value_str = getRequiredPropertyOrThrow(context,
property_name);
- if (!core::Property::StringToTime(value_str, time_value_ms, unit) ||
!core::Property::ConvertTimeUnitToMS(time_value_ms, unit, time_value_ms)) {
-throw std::runtime_error(property_name + " property is invalid: value is "
+ value_str);
- }
- return std::chrono::milliseconds(time_value_ms);
-}
-
-utils::optional getOptionalUintProperty(const core::ProcessContext&
context, const std::string& property_name) {
- uint64_t value;
- if (context.getProperty(property_name, value)) {
-return { value };
- }
- return utils::nullopt;
-}
+std::string getRequiredPropertyOrThrow(const core::ProcessContext* context,
const std::string& property_name);
Review comment:
I was under the impression that `getProperty` already throws on missing
required properties
[here](https://github.com/apache/nifi-minifi-cpp/blob/89d68e21bf28f3b25e4a64117b631668b752889f/libminifi/include/core/ConfigurableComponent.h#L227)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5020: NIFI-8435 Added Kudu Client Worker Count property
exceptionfactory commented on a change in pull request #5020:
URL: https://github.com/apache/nifi/pull/5020#discussion_r618379123
##
File path:
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/PutKudu.java
##
@@ -342,38 +346,73 @@ public void onTrigger(final ProcessContext context, final
ProcessSession session
final KerberosUser user = getKerberosUser();
if (user == null) {
-executeOnKuduClient(kuduClient -> trigger(context, session,
flowFiles, kuduClient));
+executeOnKuduClient(kuduClient -> processFlowFiles(context,
session, flowFiles, kuduClient));
return;
}
final PrivilegedExceptionAction privilegedAction = () -> {
-executeOnKuduClient(kuduClient -> trigger(context, session,
flowFiles, kuduClient));
+executeOnKuduClient(kuduClient -> processFlowFiles(context,
session, flowFiles, kuduClient));
return null;
};
final KerberosAction action = new KerberosAction<>(user,
privilegedAction, getLogger());
action.execute();
}
-private void trigger(final ProcessContext context, final ProcessSession
session, final List flowFiles, KuduClient kuduClient) throws
ProcessException {
-final RecordReaderFactory recordReaderFactory =
context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+private void processFlowFiles(final ProcessContext context, final
ProcessSession session, final List flowFiles, final KuduClient
kuduClient) {
+final Map processedRecords = new HashMap<>();
+final Map flowFileFailures = new HashMap<>();
+final Map operationFlowFileMap = new HashMap<>();
+final List pendingRowErrors = new ArrayList<>();
final KuduSession kuduSession = createKuduSession(kuduClient);
+try {
+processFlowFiles(flowFiles,
+processedRecords,
+flowFileFailures,
+operationFlowFileMap,
+pendingRowErrors,
+session,
+context,
+kuduClient,
+kuduSession);
+} finally {
+try {
+flushKuduSession(kuduSession, true, pendingRowErrors);
+} catch (final KuduException|RuntimeException e) {
+getLogger().error("KuduSession.close() Failed", e);
+}
+}
-final Map numRecords = new HashMap<>();
-final Map flowFileFailures = new HashMap<>();
-final Map operationFlowFileMap = new HashMap<>();
+if (isRollbackOnFailure() && (!pendingRowErrors.isEmpty() ||
!flowFileFailures.isEmpty())) {
+logFailures(pendingRowErrors, operationFlowFileMap);
+session.rollback();
+context.yield();
+} else {
+transferFlowFiles(flowFiles, processedRecords, flowFileFailures,
operationFlowFileMap, pendingRowErrors, session);
+}
+}
-int numBuffered = 0;
+private void processFlowFiles(final List flowFiles,
Review comment:
That seems reasonable, how about `processFlowFileRecords`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
lordgamez commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618334483
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -653,29 +690,56 @@ void TailFile::onTrigger(const
std::shared_ptr &, const st
if (!session->existsFlowFileInRelationship(Success)) {
yield();
}
+
+ first_trigger_ = false;
+}
+
+bool TailFile::isOldFileInitiallyRead(TailState ) const {
+ // This is our initial processing and no stored state was found
+ return first_trigger_ && state.checksum_ == 0;
Review comment:
Updated in 342614294774fec1e5983e70064a7b49c66dba1b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5020: NIFI-8435 Added Kudu Client Worker Count property
exceptionfactory commented on a change in pull request #5020:
URL: https://github.com/apache/nifi/pull/5020#discussion_r618371556
##
File path:
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKuduProcessor.java
##
@@ -184,10 +200,25 @@ protected KuduClient buildClient(final ProcessContext
context) {
final String masters =
context.getProperty(KUDU_MASTERS).evaluateAttributeExpressions().getValue();
final int operationTimeout =
context.getProperty(KUDU_OPERATION_TIMEOUT_MS).evaluateAttributeExpressions().asTimePeriod(TimeUnit.MILLISECONDS).intValue();
final int adminOperationTimeout =
context.getProperty(KUDU_KEEP_ALIVE_PERIOD_TIMEOUT_MS).evaluateAttributeExpressions().asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+final int workerCount = context.getProperty(WORKER_COUNT).asInteger();
+
+// Create Executor following approach of
Executors.newCachedThreadPool() using worker count as maximum pool size
+final int corePoolSize = 0;
+final long threadKeepAliveTime = 60;
+final Executor nioExecutor = new ThreadPoolExecutor(
Review comment:
Technically it isn't required as you mentioned. However, creating the
ThreadPoolExecutor instance with a custom ThreadFactory allows for created
Threads to be named using the Processor Identifier. This is useful when
troubleshooting multiple instances of PutKudu in a single VM, otherwise there
can be multiple threads with the same name, such as `kudu-nio-0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
lordgamez commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618334771
##
File path: extensions/standard-processors/processors/TailFile.h
##
@@ -147,13 +161,27 @@ class TailFile : public core::Processor {
std::string rolling_filename_pattern_;
+ std::string initial_start_position_;
Review comment:
I agree, updated in 0aa7a08cf9751f6466e0733875127d345995a57d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
lordgamez commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618335392
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -653,29 +690,56 @@ void TailFile::onTrigger(const
std::shared_ptr &, const st
if (!session->existsFlowFileInRelationship(Success)) {
yield();
}
+
+ first_trigger_ = false;
+}
+
+bool TailFile::isOldFileInitiallyRead(TailState ) const {
+ // This is our initial processing and no stored state was found
+ return first_trigger_ && state.checksum_ == 0;
}
void TailFile::processFile(const std::shared_ptr
,
const std::string _file_name,
TailState ) {
- uint64_t fsize = utils::file::FileUtils::file_size(full_file_name);
- if (fsize < state.position_) {
-processRotatedFiles(session, state);
- } else if (fsize == state.position_) {
-logger_->log_trace("Skipping file %s as its size hasn't change since last
read", state.file_name_);
-return;
+ if (isOldFileInitiallyRead(state)) {
+if (initial_start_position_ == "Beginning of Time") {
+ processAllRotatedFiles(session, state);
+} else if (initial_start_position_ == "Current Time") {
+ state.position_ = utils::file::FileUtils::file_size(full_file_name);
+ state.last_read_time_ = std::chrono::system_clock::now();
Review comment:
Good point, I added this use case as a separate test and added the fix
in 1eaf3d3752352a8c6210a8df127c71d9ec60debd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1055: MINIFICPP-1454 Reduce duplication of CMake parameters in docker build
adamdebreceni commented on a change in pull request #1055:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1055#discussion_r618334196
##
File path: cmake/DockerConfig.cmake
##
@@ -22,7 +22,6 @@ add_custom_target(
-u 1000
-g 1000
-v
${PROJECT_VERSION_MAJOR}.${PROJECT_VERSION_MINOR}.${PROJECT_VERSION_PATCH}
--c IMAGE_TYPE=release
Review comment:
should we use `-i release` here as other targets use it like that? or
should we remove `-i, -u, -g` wherever they align with the defaults in
`DockerBuild.sh`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
fgerlits commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618353909
##
File path: extensions/standard-processors/processors/TailFile.cpp
##
@@ -653,29 +690,56 @@ void TailFile::onTrigger(const
std::shared_ptr &, const st
if (!session->existsFlowFileInRelationship(Success)) {
yield();
}
+
+ first_trigger_ = false;
+}
+
+bool TailFile::isOldFileInitiallyRead(TailState ) const {
+ // This is our initial processing and no stored state was found
+ return first_trigger_ && state.checksum_ == 0;
}
void TailFile::processFile(const std::shared_ptr
,
const std::string _file_name,
TailState ) {
- uint64_t fsize = utils::file::FileUtils::file_size(full_file_name);
- if (fsize < state.position_) {
-processRotatedFiles(session, state);
- } else if (fsize == state.position_) {
-logger_->log_trace("Skipping file %s as its size hasn't change since last
read", state.file_name_);
-return;
+ if (isOldFileInitiallyRead(state)) {
+if (initial_start_position_ == "Beginning of Time") {
+ processAllRotatedFiles(session, state);
+} else if (initial_start_position_ == "Current Time") {
+ state.position_ = utils::file::FileUtils::file_size(full_file_name);
+ state.last_read_time_ = std::chrono::system_clock::now();
Review comment:
I would use `utils::file::FileUtils::computeChecksum()` as it is done in
`findRotatedFiles()` (line 656), mainly for consistency, but it's also slightly
shorter/simpler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1055: MINIFICPP-1454 Reduce duplication of CMake parameters in docker build
adamdebreceni commented on a change in pull request #1055:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1055#discussion_r618334196
##
File path: cmake/DockerConfig.cmake
##
@@ -22,7 +22,6 @@ add_custom_target(
-u 1000
-g 1000
-v
${PROJECT_VERSION_MAJOR}.${PROJECT_VERSION_MINOR}.${PROJECT_VERSION_PATCH}
--c IMAGE_TYPE=release
Review comment:
should we use `-i release` here as well (other targets use it like
that)? or should we remove `-i, -u, -g` wherever they align with the defaults
in `DockerBuild.sh`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1052: MINIFICPP-1244 Support the Initial Start Position property in TailFile
lordgamez commented on a change in pull request #1052:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1052#discussion_r618335630
##
File path: extensions/standard-processors/tests/unit/TailFileTests.cpp
##
@@ -1536,3 +1514,197 @@ TEST_CASE("TailFile interprets the lookup frequency
property correctly", "[multi
REQUIRE(LogTestController::getInstance().contains("Logged 1 flow files"));
}
}
+
+TEST_CASE("TailFile reads from a single file when Initial Start Position is
set", "[initialStartPosition]") {
+ TestController testController;
+ LogTestController::getInstance().setTrace();
+
LogTestController::getInstance().setDebug();
+
+ std::shared_ptr plan = testController.createPlan();
+ std::shared_ptr tailfile = plan->addProcessor("TailFile",
"tailfileProc");
+ std::shared_ptr logattribute =
plan->addProcessor("LogAttribute", "logattribute",
core::Relationship("success", "description"), true);
+
+ auto dir = minifi::utils::createTempDir();
+ createTempFile(dir, ROLLED_OVER_TMP_FILE, ROLLED_OVER_TAIL_DATA);
+ auto temp_file_path = createTempFile(dir, TMP_FILE, NEWLINE_FILE);
+
+ plan->setProperty(logattribute,
org::apache::nifi::minifi::processors::LogAttribute::FlowFilesToLog.getName(),
"0");
+ plan->setProperty(tailfile,
org::apache::nifi::minifi::processors::TailFile::FileName.getName(),
temp_file_path);
+ plan->setProperty(tailfile,
org::apache::nifi::minifi::processors::TailFile::Delimiter.getName(), "\n");
+
+ SECTION("Initial Start Position is set to Beginning of File") {
+plan->setProperty(tailfile,
org::apache::nifi::minifi::processors::TailFile::InitialStartPosition.getName(),
"Beginning of File");
+
+testController.runSession(plan);
+
+REQUIRE(LogTestController::getInstance().contains("Logged 1 flow files"));
+REQUIRE(LogTestController::getInstance().contains("Size:" +
std::to_string(NEWLINE_FILE.find_first_of('\n') + 1) + " Offset:0"));
+
+plan->reset(true);
+
LogTestController::getInstance().resetStream(LogTestController::getInstance().log_output);
+
+appendTempFile(dir, TMP_FILE, NEW_TAIL_DATA);
+
+testController.runSession(plan);
+
+REQUIRE(LogTestController::getInstance().contains("Logged 1 flow files"));
+REQUIRE(LogTestController::getInstance().contains("Size:" +
std::to_string(NEWLINE_FILE.size() - NEWLINE_FILE.find_first_of('\n') +
NEW_TAIL_DATA.find_first_of('\n')) + " Offset:0"));
+ }
+
+ SECTION("Initial Start Position is set to Beginning of Time") {
+plan->setProperty(tailfile,
org::apache::nifi::minifi::processors::TailFile::InitialStartPosition.getName(),
"Beginning of Time");
+
+testController.runSession(plan);
+
+REQUIRE(LogTestController::getInstance().contains("Logged 2 flow files"));
+REQUIRE(LogTestController::getInstance().contains("Size:" +
std::to_string(NEWLINE_FILE.find_first_of('\n') + 1) + " Offset:0"));
+REQUIRE(LogTestController::getInstance().contains("Size:" +
std::to_string(ROLLED_OVER_TAIL_DATA.find_first_of('\n') + 1) + " Offset:0"));
+
+plan->reset(true);
+
LogTestController::getInstance().resetStream(LogTestController::getInstance().log_output);
+
+appendTempFile(dir, TMP_FILE, NEW_TAIL_DATA);
+
+testController.runSession(plan);
+
+REQUIRE(LogTestController::getInstance().contains("Logged 1 flow files"));
+REQUIRE(LogTestController::getInstance().contains("Size:" +
std::to_string(NEWLINE_FILE.size() - NEWLINE_FILE.find_first_of('\n') +
NEW_TAIL_DATA.find_first_of('\n')) + " Offset:0"));
+ }
+
+ SECTION("Initial Start Position is set to Current Time") {
+plan->setProperty(tailfile,
org::apache::nifi::minifi::processors::TailFile::InitialStartPosition.getName(),
"Current Time");
+
+testController.runSession(plan);
+
+REQUIRE(LogTestController::getInstance().contains("Logged 0 flow files"));
+
+plan->reset(true);
+
LogTestController::getInstance().resetStream(LogTestController::getInstance().log_output);
+
+appendTempFile(dir, TMP_FILE, NEW_TAIL_DATA);
+
+testController.runSession(plan);
+
+REQUIRE(LogTestController::getInstance().contains("Logged 1 flow files"));
+REQUIRE(LogTestController::getInstance().contains("Size:" +
std::to_string(NEW_TAIL_DATA.find_first_of('\n') + 1) + " Offset:0"));
+ }
+
+ LogTestController::getInstance().reset();
+}
+
+TEST_CASE("TailFile reads multiple files when Initial Start Position is set",
"[initialStartPosition]") {
+ TestController testController;
+ LogTestController::getInstance().setTrace();
+
LogTestController::getInstance().setDebug();
+
+ std::shared_ptr plan = testController.createPlan();
+ std::shared_ptr tailfile = plan->addProcessor("TailFile",
"tailfileProc");
+ std::shared_ptr logattribute =
plan->addProcessor("LogAttribute", "logattribute",
core::Relationship("success", "description"), true);
+
+ auto dir = minifi::utils::createTempDir();
+ createTempFile(dir, ROLLED_OVER_TMP_FILE, ROLLED_OVER_TAIL_DATA);
+
