[GitHub] [nifi] mattyb149 commented on pull request #6391: NIFI-9402: Adding DatabaseParameterProvider

[GitBox]

mattyb149 commented on PR #6391:
URL: https://github.com/apache/nifi/pull/6391#issuecomment-1258943926

   Reviewing...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] thenatog commented on pull request #6394: NIFI-9637: Adding GcpSecretManagerParameterProvider

[GitBox]

thenatog commented on PR #6394:
URL: https://github.com/apache/nifi/pull/6394#issuecomment-1258887013

   Will review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-10542) Parameter Providers UI improvements

[Matt Gilman (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-10542:
---
Fix Version/s: 1.18.0

> Parameter Providers UI improvements
> ---
>
> Key: NIFI-10542
> URL: https://issues.apache.org/jira/browse/NIFI-10542
> Project: Apache NiFi
>  Issue Type: Sub-task
>Affects Versions: 1.16.3
>Reporter: Margot Tien
>Assignee: Margot Tien
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Additional UI improvements for the recent introduction of Parameter Providers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (NIFI-10542) Parameter Providers UI improvements

[Matt Gilman (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-10542:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Parameter Providers UI improvements
> ---
>
> Key: NIFI-10542
> URL: https://issues.apache.org/jira/browse/NIFI-10542
> Project: Apache NiFi
>  Issue Type: Sub-task
>Affects Versions: 1.16.3
>Reporter: Margot Tien
>Assignee: Margot Tien
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Additional UI improvements for the recent introduction of Parameter Providers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10542) Parameter Providers UI improvements

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609734#comment-17609734
 ] 

ASF subversion and git services commented on NIFI-10542:


Commit 34084d083acac205076840a41a859849a3d71471 in nifi's branch 
refs/heads/main from M Tien
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=34084d083a ]

NIFI-10542: set sensitivity for new parameters in a synced parameter context 
(#6446)

* NIFI-10542: set sensitivity for new parameters in a synced parameter context

* show removed or missing but referenced parameters with tooltip in the 
selectable parameters table
- remove default parameter value

* address review feedback
- fix showing a referencing icon for a missing but removed parameter

* fix discovering new parameters

This closes #6446 

> Parameter Providers UI improvements
> ---
>
> Key: NIFI-10542
> URL: https://issues.apache.org/jira/browse/NIFI-10542
> Project: Apache NiFi
>  Issue Type: Sub-task
>Affects Versions: 1.16.3
>Reporter: Margot Tien
>Assignee: Margot Tien
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Additional UI improvements for the recent introduction of Parameter Providers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] mcgilman merged pull request #6446: NIFI-10542: set sensitivity for new parameters in a synced parameter context

[GitBox]

mcgilman merged PR #6446:
URL: https://github.com/apache/nifi/pull/6446


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] joewitt commented on pull request #6256: [NIFI-10289] PutDatabaseRecord generating incomplete Merge SQL when using Oracle12DatabaseAdapter

[GitBox]

joewitt commented on PR #6256:
URL: https://github.com/apache/nifi/pull/6256#issuecomment-1258738827

   This PR is closed as it was merged to main and JIRA closed.  If we want to 
make changes lets open a new PR.  Thanks.  Getting things pulled togehter for 
RC1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] joewitt closed pull request #6256: [NIFI-10289] PutDatabaseRecord generating incomplete Merge SQL when using Oracle12DatabaseAdapter

[GitBox]

joewitt closed pull request #6256: [NIFI-10289] PutDatabaseRecord generating 
incomplete Merge SQL when using Oracle12DatabaseAdapter
URL: https://github.com/apache/nifi/pull/6256


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] eduardofontes opened a new pull request, #6256: [NIFI-10289] PutDatabaseRecord generating incomplete Merge SQL when using Oracle12DatabaseAdapter

[GitBox]

eduardofontes opened a new pull request, #6256:
URL: https://github.com/apache/nifi/pull/6256

   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   
   [NIFI-10289](https://issues.apache.org/jira/browse/NIFI-10289)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [ ] Pull Request based on current revision of the `main` branch
   - [ ] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [ ] JDK 8
 - [ ] JDK 11
 - [ ] JDK 17
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-10289) PutDatabaseRecord generating incomplete Merge SQL when using Oracle12DatabaseAdapter

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-10289:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> PutDatabaseRecord generating incomplete Merge SQL when using 
> Oracle12DatabaseAdapter
> 
>
> Key: NIFI-10289
> URL: https://issues.apache.org/jira/browse/NIFI-10289
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.16.3
>Reporter: Eduardo Mota Fontes
>Assignee: Eduardo Mota Fontes
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Processor PutDatabaseRecord is generating incomplete SQL statement when MERGE 
> statement type for ORACLE12+.
> It isn't considering all Update Keys.
> I'm getting this SQL:
> {code:sql}
> MERGE INTO table USING (SELECT ? CASO, ? REFERENCIA_NOME, ? 
> CASO_REFERENCIA_INDICA, 1 REFERENCIA_CODIGO FROM DUAL) n ON (table.CASO = 
> n.CASO) WHEN NOT MATCHED THEN INSERT (CASO, REFERENCIA_NOME, 
> CASO_REFERENCIA_INDICA, REFERENCIA_CODIGO) VALUES (n.CASO, n.REFERENCIA_NOME, 
> n.CASO_REFERENCIA_INDICA, n.REFERENCIA_CODIGO) WHEN MATCHED THEN UPDATE SET 
> table.REFERENCIA_NOME = n.REFERENCIA_NOME, table.CASO_REFERENCIA_INDICA = 
> n.CASO_REFERENCIA_INDICA, table.REFERENCIA_CODIGO = n.REFERENCIA_CODIGO{code}
> Should be:
> {code:sql}
> MERGE INTO table USING (SELECT ? CASO, ? REFERENCIA_NOME, ? 
> CASO_REFERENCIA_INDICA, 1 REFERENCIA_CODIGO FROM DUAL) n ON (table.CASO = 
> n.CASO AND table.REFERENCIA_CODIGO = n.REFERENCIA_CODIGO) WHEN NOT MATCHED 
> THEN INSERT (CASO, REFERENCIA_NOME, CASO_REFERENCIA_INDICA, 
> REFERENCIA_CODIGO) VALUES (n.CASO, n.REFERENCIA_NOME, 
> n.CASO_REFERENCIA_INDICA, n.REFERENCIA_CODIGO) WHEN MATCHED THEN UPDATE SET 
> table.REFERENCIA_NOME = n.REFERENCIA_NOME, table.CASO_REFERENCIA_INDICA = 
> n.CASO_REFERENCIA_INDICA{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10289) PutDatabaseRecord generating incomplete Merge SQL when using Oracle12DatabaseAdapter

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609708#comment-17609708
 ] 

ASF subversion and git services commented on NIFI-10289:


Commit 9bc63920aad21bd1bba2c1f4c6b5f050081459b1 in nifi's branch 
refs/heads/main from Eduardo Fontes
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9bc63920aa ]

NIFI-10289 This closes #6256. Change getConflictColumnsClause
NIFI-10289 Test upsert to catch this issue

Signed-off-by: Joe Witt 


> PutDatabaseRecord generating incomplete Merge SQL when using 
> Oracle12DatabaseAdapter
> 
>
> Key: NIFI-10289
> URL: https://issues.apache.org/jira/browse/NIFI-10289
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.16.3
>Reporter: Eduardo Mota Fontes
>Assignee: Eduardo Mota Fontes
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Processor PutDatabaseRecord is generating incomplete SQL statement when MERGE 
> statement type for ORACLE12+.
> It isn't considering all Update Keys.
> I'm getting this SQL:
> {code:sql}
> MERGE INTO table USING (SELECT ? CASO, ? REFERENCIA_NOME, ? 
> CASO_REFERENCIA_INDICA, 1 REFERENCIA_CODIGO FROM DUAL) n ON (table.CASO = 
> n.CASO) WHEN NOT MATCHED THEN INSERT (CASO, REFERENCIA_NOME, 
> CASO_REFERENCIA_INDICA, REFERENCIA_CODIGO) VALUES (n.CASO, n.REFERENCIA_NOME, 
> n.CASO_REFERENCIA_INDICA, n.REFERENCIA_CODIGO) WHEN MATCHED THEN UPDATE SET 
> table.REFERENCIA_NOME = n.REFERENCIA_NOME, table.CASO_REFERENCIA_INDICA = 
> n.CASO_REFERENCIA_INDICA, table.REFERENCIA_CODIGO = n.REFERENCIA_CODIGO{code}
> Should be:
> {code:sql}
> MERGE INTO table USING (SELECT ? CASO, ? REFERENCIA_NOME, ? 
> CASO_REFERENCIA_INDICA, 1 REFERENCIA_CODIGO FROM DUAL) n ON (table.CASO = 
> n.CASO AND table.REFERENCIA_CODIGO = n.REFERENCIA_CODIGO) WHEN NOT MATCHED 
> THEN INSERT (CASO, REFERENCIA_NOME, CASO_REFERENCIA_INDICA, 
> REFERENCIA_CODIGO) VALUES (n.CASO, n.REFERENCIA_NOME, 
> n.CASO_REFERENCIA_INDICA, n.REFERENCIA_CODIGO) WHEN MATCHED THEN UPDATE SET 
> table.REFERENCIA_NOME = n.REFERENCIA_NOME, table.CASO_REFERENCIA_INDICA = 
> n.CASO_REFERENCIA_INDICA{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] asfgit closed pull request #6256: [NIFI-10289] PutDatabaseRecord generating incomplete Merge SQL when using Oracle12DatabaseAdapter

[GitBox]

asfgit closed pull request #6256: [NIFI-10289] PutDatabaseRecord generating 
incomplete Merge SQL when using Oracle12DatabaseAdapter
URL: https://github.com/apache/nifi/pull/6256


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (NIFI-10442) Create PutIceberg processor

[Joe Witt (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609707#comment-17609707
 ] 

Joe Witt commented on NIFI-10442:
-

Went to remove fix version but there is a ton of review/iteration going on so 
will give it another day or so to see if it makes it in.  If necessary will 
remove fix version and get the RC going.

> Create PutIceberg processor
> ---
>
> Key: NIFI-10442
> URL: https://issues.apache.org/jira/browse/NIFI-10442
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Mark Bathori
>Assignee: Mark Bathori
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Add a processor that is able to ingest data into Iceberg tables.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (NIFI-10442) Create PutIceberg processor

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-10442:

Fix Version/s: 1.18.0

> Create PutIceberg processor
> ---
>
> Key: NIFI-10442
> URL: https://issues.apache.org/jira/browse/NIFI-10442
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Mark Bathori
>Assignee: Mark Bathori
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Add a processor that is able to ingest data into Iceberg tables.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (NIFI-10442) Create PutIceberg processor

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-10442:

Fix Version/s: (was: 1.18.0)

> Create PutIceberg processor
> ---
>
> Key: NIFI-10442
> URL: https://issues.apache.org/jira/browse/NIFI-10442
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Mark Bathori
>Assignee: Mark Bathori
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Add a processor that is able to ingest data into Iceberg tables.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (NIFI-10473) Parameter Provider Fetch REST call authorization check is too restrictive

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-10473.
-
Resolution: Fixed

appears this was merged with the work for parameter providers

> Parameter Provider Fetch REST call authorization check is too restrictive
> -
>
> Key: NIFI-10473
> URL: https://issues.apache.org/jira/browse/NIFI-10473
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Joe Gresock
>Assignee: Joe Gresock
>Priority: Minor
> Fix For: 1.18.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When bringing up the Fetch Parameters dialog, if the user is not authorized 
> on any referencing component, the dialog fails to load.  This is overly 
> restrictive, as NiFi already prevents applying the parameters in this 
> scenario.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] turcsanyip commented on a diff in pull request #6430: NIFI-10523 - Udated Google Drive processor documentations.

[GitBox]

turcsanyip commented on code in PR #6430:
URL: https://github.com/apache/nifi/pull/6430#discussion_r980521511


##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/resources/docs/org.apache.nifi.processors.gcp.drive.ListGoogleDrive/additionalDetails.html:
##
@@ -0,0 +1,51 @@
+
+http://www.w3.org/1999/html";>
+
+
+
+
+ListGoogleDrive
+
+
+
+
+Accessing Google Drive from NiFi
+
+
+This processor uses Google Cloud credentials for authentication to access 
Google Drive.
+The following steps are required to prepare the Google Cloud and Google 
Drive accounts for the processors:
+
+
+Enable Google Drive API in Google Cloud - Follow instructions at
+https://developers.google.com/workspace/guides/enable-apis

Review Comment:
   ```suggestion
   https://developers.google.com/workspace/guides/enable-apis";>https://developers.google.com/workspace/guides/enable-apis
   ```



##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/resources/docs/org.apache.nifi.processors.gcp.drive.ListGoogleDrive/additionalDetails.html:
##
@@ -0,0 +1,51 @@
+
+http://www.w3.org/1999/html";>
+
+
+
+
+ListGoogleDrive
+
+
+
+
+Accessing Google Drive from NiFi
+
+
+This processor uses Google Cloud credentials for authentication to access 
Google Drive.
+The following steps are required to prepare the Google Cloud and Google 
Drive accounts for the processors:
+
+
+Enable Google Drive API in Google Cloud - Follow instructions at
+https://developers.google.com/workspace/guides/enable-apis
+and search for 'Google Drive API'.
+
+Grant access to Google Drive folder - In Google Cloud Console 
navigate to IAM & Admin -> Service
+Accounts. Take a note of the email of the service account you are 
going to use.
+Navigate to the folder to be listed in Google Drive. Right-click on 
the Folder -> Share. Enter the
+service account email.
+
+Find Folder ID - Enter the folder to be listed. The URL in your 
browser will include the ID at the end of
+the URL.

Review Comment:
   Maybe it is more straightforward where to enter:
   ```suggestion
   Find Folder ID - Navigate to the folder to be listed in 
Google Drive and enter it. The URL in your browser will include the ID at the 
end of
   the URL.
   ```



##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/resources/docs/org.apache.nifi.processors.gcp.drive.FetchGoogleDrive/additionalDetails.html:
##
@@ -0,0 +1,44 @@
+
+http://www.w3.org/1999/html";>
+
+
+
+
+FetchGoogleDrive
+
+
+
+
+Accessing Google Drive from NiFi
+
+
+This processor uses Google Cloud credentials for authentication to access 
Google Drive.
+The following steps are required to prepare the Google Cloud and Google 
Drive accounts for the processors:
+
+
+Enable Google Drive API in Google Cloud - Follow instructions at
+https://developers.google.com/workspace/guides/enable-apis

Review Comment:
   ```suggestion
   https://developers.google.com/workspace/guides/enable-apis";>https://developers.google.com/workspace/guides/enable-apis
   ```



##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/resources/docs/org.apache.nifi.processors.gcp.drive.ListGoogleDrive/additionalDetails.html:
##
@@ -0,0 +1,51 @@
+
+http://www.w3.org/1999/html";>
+
+
+
+
+ListGoogleDrive
+
+
+
+
+Accessing Google Drive from NiFi
+
+
+This processor uses Google Cloud credentials for authentication to access 
Google Drive.
+The following steps are required to prepare the Google Cloud and Google 
Drive accounts for the processors:
+
+
+Enable Google Drive API in Google Cloud - Follow instructions at
+https://developers.google.com/workspace/guides/enable-apis
+and search for 'Google Drive API'.
+
+Grant access to Google Drive folder - In Google Cloud Console 
navigate to IAM & Admin -> Service
+Accounts. Take a note of the email of the service account you are 
going to use.
+Navigate to the folder to be listed in Google Drive. Right-click on 
the Folder -> Share. Enter the
+service account email.
+
+Find Folder ID - Enter the folder to be listed. The URL in your 
browser will include the ID at the end of
+the URL.
+For example, if the URL were 
https://drive.google.com/drive/folders/1trTraPVCnX5_TNwO8d9P_bz278xWOmGm, the
+Folder ID would be 1trTraPVCnX5_TNwO8d9P_bz278xWOmGm

Review Comment:
   ```suggestion
   For example, if the URL were 
https://drive.google.com/drive/folders/1trTraPVCnX5_TNwO8d9P_bz278xWOmGm,
 the
   Folder ID would be 1trTraPVCnX5_TNwO8d9P_bz278xWOmGm
   ```



##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/resources/docs/org.apache.nifi.processors.gcp.drive.ListGoogleDrive/additionalDetails.html:
##
@@ -0,0 

[jira] [Assigned] (NIFI-10288) ListS3 processor: Better describe forward slash requirements for Prefix property

[Andrew M. Lim (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew M. Lim reassigned NIFI-10288:


Assignee: Andrew M. Lim

> ListS3 processor: Better describe forward slash requirements for Prefix 
> property
> 
>
> Key: NIFI-10288
> URL: https://issues.apache.org/jira/browse/NIFI-10288
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Andrew M. Lim
>Assignee: Andrew M. Lim
>Priority: Minor
> Attachments: image-2022-07-27-14-41-44-792.png
>
>
> The current help text for the property:
>  !image-2022-07-27-14-41-44-792.png! 
> Based on my testing, ListS3 works whether or not the prefix has a trailing 
> forward slash ('/'). However, ListS3 does not work if the prefix has a 
> preceeding forward slash.
> Suggest the following update:
> The prefix used to filter the object list. Do not start with a forward slash 
> ('/').



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (NIFI-10532) PutFTP does not group the batch by user name

[Mark Payne (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne resolved NIFI-10532.
---
Resolution: Fixed

> PutFTP does not group the batch by user name
> 
>
> Key: NIFI-10532
> URL: https://issues.apache.org/jira/browse/NIFI-10532
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.17.0, 1.16.1
>Reporter: Christoph Langheld
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 1.18.0
>
> Attachments: 01-PutFTP-ProcessGroup.png, 02-PutFTP-Setting.png, 
> 03-PutFTP-Result.png, PutFTP-Bug.xml
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Hello,
> for the PutFTP processor we set the host, name, password, port, and target 
> directory dynamically via UpdateAttribute.
> We now have the problem, that the PutFTP processor transmits every file with 
> the same user even the ftp user name changed. The target host does not 
> change, only the user.
> To reproduce I attached a process group as template ([^PutFTP-Bug.xml]). You 
> have to adapt the ftp server settings within the UpdateAttribute processor to 
> your environment.
> The process group generates 50 flow files.
> UpdateAttribute sets the ftp user credentials and sets a filename prefix 
> (NIFI-A_ respectively NIFI-B_).
> If everything would work correctly all files with prefix NIFI-A_ should be 
> transfered to the ftp server as user nifi-a and the rest as user nifi-b.
> But every file will be transferred as the same user (nifi-a).
> PutFTP should group the batch by host, login credentials (user, password, 
> port) and target directory.
> !01-PutFTP-ProcessGroup.png|width=639,height=607!
> PutFTP settings:
> !02-PutFTP-Setting.png|width=750,height=523!
> Result on the ftp server (logged in as user nifi-a):
> !03-PutFTP-Result.png|width=268,height=482!
>  
> Thank you and regards
> Christoph
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10532) PutFTP does not group the batch by user name

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609681#comment-17609681
 ] 

ASF subversion and git services commented on NIFI-10532:


Commit 9e6ee656b54a91384ce2d2d31cd2d0489534f61b in nifi's branch 
refs/heads/main from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9e6ee656b5 ]

NIFI-10532 ensuring client gets reset if any of the key values 
host/port/user/pw change on a per ff basis (#6445)



> PutFTP does not group the batch by user name
> 
>
> Key: NIFI-10532
> URL: https://issues.apache.org/jira/browse/NIFI-10532
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.17.0, 1.16.1
>Reporter: Christoph Langheld
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 1.18.0
>
> Attachments: 01-PutFTP-ProcessGroup.png, 02-PutFTP-Setting.png, 
> 03-PutFTP-Result.png, PutFTP-Bug.xml
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Hello,
> for the PutFTP processor we set the host, name, password, port, and target 
> directory dynamically via UpdateAttribute.
> We now have the problem, that the PutFTP processor transmits every file with 
> the same user even the ftp user name changed. The target host does not 
> change, only the user.
> To reproduce I attached a process group as template ([^PutFTP-Bug.xml]). You 
> have to adapt the ftp server settings within the UpdateAttribute processor to 
> your environment.
> The process group generates 50 flow files.
> UpdateAttribute sets the ftp user credentials and sets a filename prefix 
> (NIFI-A_ respectively NIFI-B_).
> If everything would work correctly all files with prefix NIFI-A_ should be 
> transfered to the ftp server as user nifi-a and the rest as user nifi-b.
> But every file will be transferred as the same user (nifi-a).
> PutFTP should group the batch by host, login credentials (user, password, 
> port) and target directory.
> !01-PutFTP-ProcessGroup.png|width=639,height=607!
> PutFTP settings:
> !02-PutFTP-Setting.png|width=750,height=523!
> Result on the ftp server (logged in as user nifi-a):
> !03-PutFTP-Result.png|width=268,height=482!
>  
> Thank you and regards
> Christoph
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] markap14 merged pull request #6445: NIFI-10532 ensuring client gets reset if any of the key values host/p…

[GitBox]

markap14 merged PR #6445:
URL: https://github.com/apache/nifi/pull/6445


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] markap14 commented on pull request #6445: NIFI-10532 ensuring client gets reset if any of the key values host/p…

[GitBox]

markap14 commented on PR #6445:
URL: https://github.com/apache/nifi/pull/6445#issuecomment-1258626372

   Thanks @joewitt for the fix. And to @exceptionfactory and @arpadboda for 
reviewing as well. Code changes make sense (though I agree it would make sense 
to cleanup in the future). Was able to do some testing. All looks good to me as 
well. Will merge to main.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-10542) Parameter Providers UI improvements

[Margot Tien (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Margot Tien updated NIFI-10542:
---
Status: Patch Available  (was: In Progress)

> Parameter Providers UI improvements
> ---
>
> Key: NIFI-10542
> URL: https://issues.apache.org/jira/browse/NIFI-10542
> Project: Apache NiFi
>  Issue Type: Sub-task
>Affects Versions: 1.16.3
>Reporter: Margot Tien
>Assignee: Margot Tien
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Additional UI improvements for the recent introduction of Parameter Providers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (NIFI-9514) Add UI support for Parameter Providers in Controller Settings

[Andrew M. Lim (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew M. Lim updated NIFI-9514:

Description: 
Update the UI to allow configuring Parameter Providers.
 * In Controller Settings, add a new tab, "PARAMETER PROVIDERS", that shows the 
parameter providers listing
 * Include a "+" button that opens an "Add Parameter Provider" dialog
 * In the listing, include a "Fetch Parameters" button next to each Parameter 
Provider, which would cause it to pull all available parameters and values from 
the external source, based on the current configuration
 * Update the "REPORTING TASK CONTROLLER SERVICES" tab name to "-CONTROLLER 
LEVEL- MANAGEMENT CONTROLLER SERVICES"

  was:
Update the UI to allow configuring Parameter Providers.
 * In Controller Settings, add a new tab, "PARAMETER PROVIDERS", that shows the 
parameter providers listing
 * Include a "+" button that opens an "Add Parameter Provider" dialog
 * In the listing, include a "Fetch Parameters" button next to each Parameter 
Provider, which would cause it to pull all available parameters and values from 
the external source, based on the current configuration
 * Update the "REPORTING TASK CONTROLLER SERVICES" tab name to "CONTROLLER 
LEVEL CONTROLLER SERVICES"


> Add UI support for Parameter Providers in Controller Settings
> -
>
> Key: NIFI-9514
> URL: https://issues.apache.org/jira/browse/NIFI-9514
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.16.3
>Reporter: Margot Tien
>Assignee: Margot Tien
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Update the UI to allow configuring Parameter Providers.
>  * In Controller Settings, add a new tab, "PARAMETER PROVIDERS", that shows 
> the parameter providers listing
>  * Include a "+" button that opens an "Add Parameter Provider" dialog
>  * In the listing, include a "Fetch Parameters" button next to each Parameter 
> Provider, which would cause it to pull all available parameters and values 
> from the external source, based on the current configuration
>  * Update the "REPORTING TASK CONTROLLER SERVICES" tab name to "-CONTROLLER 
> LEVEL- MANAGEMENT CONTROLLER SERVICES"



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] mtien-apache commented on a diff in pull request #6446: NIFI-10542: set sensitivity for new parameters in a synced parameter context

[GitBox]

mtien-apache commented on code in PR #6446:
URL: https://github.com/apache/nifi/pull/6446#discussion_r980484137


##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-parameter-provider.js:
##
@@ -2383,7 +2439,7 @@
 
 if 
(!_.isEmpty(dataContext.parameterStatus.parameter.parameter.referencingComponents))
 {
 // disable checkboxes
-if (dataContext.sensitivity === SENSITIVE) {
+if (dataContext.sensitivity === SENSITIVE || 
dataContext.status === 'REMOVED' || dataContext.status === 
'MISSING_BUT_REFERENCED') {

Review Comment:
   Good catch. No, we don't need the check anymore. It was a leftover change 
before I decided to check the status before this line. Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] mcgilman commented on a diff in pull request #6446: NIFI-10542: set sensitivity for new parameters in a synced parameter context

[GitBox]

mcgilman commented on code in PR #6446:
URL: https://github.com/apache/nifi/pull/6446#discussion_r980475137


##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-parameter-provider.js:
##
@@ -2383,7 +2439,7 @@
 
 if 
(!_.isEmpty(dataContext.parameterStatus.parameter.parameter.referencingComponents))
 {
 // disable checkboxes
-if (dataContext.sensitivity === SENSITIVE) {
+if (dataContext.sensitivity === SENSITIVE || 
dataContext.status === 'REMOVED' || dataContext.status === 
'MISSING_BUT_REFERENCED') {

Review Comment:
   Are these conditions needed? The check above should cover when the status is 
`REMOVED` or `MISSING_BUT_REFERENCED`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-10550) Fixing SSL context service validation for NifiRegistryFlowRegistryClient

[Mark Payne (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-10550:
--
Affects Version/s: 1.18.0

> Fixing SSL context service validation for NifiRegistryFlowRegistryClient
> 
>
> Key: NIFI-10550
> URL: https://issues.apache.org/jira/browse/NIFI-10550
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.18.0
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The validation of the SSL Context Service within 
> NifiRegistryFlowRegistryClient should trigger when the Controller Service is 
> set but only the Trust of Key store is filled not both. The current logit 
> triggers a validation error when both are set instead, this should be fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10550) Fixing SSL context service validation for NifiRegistryFlowRegistryClient

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609676#comment-17609676
 ] 

ASF subversion and git services commented on NIFI-10550:


Commit e5059e0514bf18b854fd833d3c5350d0b1a956ba in nifi's branch 
refs/heads/main from simonbence
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=e5059e0514 ]

NIFI-10550 Fixing SSH Context Service validation for 
NifiRegistryFlowRegistryClient (#6451)



> Fixing SSL context service validation for NifiRegistryFlowRegistryClient
> 
>
> Key: NIFI-10550
> URL: https://issues.apache.org/jira/browse/NIFI-10550
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The validation of the SSL Context Service within 
> NifiRegistryFlowRegistryClient should trigger when the Controller Service is 
> set but only the Trust of Key store is filled not both. The current logit 
> triggers a validation error when both are set instead, this should be fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (NIFI-10497) Making RegistryClient an extension point

[Mark Payne (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne resolved NIFI-10497.
---
Resolution: Fixed

> Making RegistryClient an extension point
> 
>
> Key: NIFI-10497
> URL: https://issues.apache.org/jira/browse/NIFI-10497
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Critical
> Fix For: 1.18.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Currently NiFi is capable to connect only to the NiFi Registry as a "registry 
> repository". It would be beneficial to give NiFi the capability to depend on 
> other services.
> In order to this, I suggest to decouple the registry behaviour from the NiFi 
> Registry as much as it is possible (inlcuding but not only the API and the 
> Resources) and move the actual implementation behind this new API.
> To be able to add new implementations, this new API must be an extension 
> point applying the usual NiFi instruments for this. Also it is paramunt to 
> keep the continuity with the current usages and make the implementation 
> capable to process the current REST call format. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (NIFI-10550) Fixing SSL context service validation for NifiRegistryFlowRegistryClient

[Mark Payne (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne resolved NIFI-10550.
---
Fix Version/s: 1.18.0
   Resolution: Fixed

> Fixing SSL context service validation for NifiRegistryFlowRegistryClient
> 
>
> Key: NIFI-10550
> URL: https://issues.apache.org/jira/browse/NIFI-10550
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The validation of the SSL Context Service within 
> NifiRegistryFlowRegistryClient should trigger when the Controller Service is 
> set but only the Trust of Key store is filled not both. The current logit 
> triggers a validation error when both are set instead, this should be fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] markap14 merged pull request #6451: NIFI-10550 Fixing SSH Context Service validation for NifiRegistryFlowRegistryClient

[GitBox]

markap14 merged PR #6451:
URL: https://github.com/apache/nifi/pull/6451


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] markap14 commented on pull request #6451: NIFI-10550 Fixing SSH Context Service validation for NifiRegistryFlowRegistryClient

[GitBox]

markap14 commented on PR #6451:
URL: https://github.com/apache/nifi/pull/6451#issuecomment-1258592187

   Thanks @simonbence good catch. Now that the UI has been integrated I can 
easily verify that this fixes the issue. +1 will merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (NIFI-10538) UI changes to support RegsitryClient as an extension point

[Matt Gilman (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman resolved NIFI-10538.

Resolution: Fixed

> UI changes to support RegsitryClient as an extension point
> --
>
> Key: NIFI-10538
> URL: https://issues.apache.org/jira/browse/NIFI-10538
> Project: Apache NiFi
>  Issue Type: Sub-task
>Reporter: Shane Ardell
>Assignee: Shane Ardell
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] mcgilman merged pull request #6437: NIFI-10538: UI updates for making RegistryClient an extension point

[GitBox]

mcgilman merged PR #6437:
URL: https://github.com/apache/nifi/pull/6437


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (NIFI-10538) UI changes to support RegsitryClient as an extension point

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609674#comment-17609674
 ] 

ASF subversion and git services commented on NIFI-10538:


Commit 17084a1f0984af1cc961828d054e0adfa1949969 in nifi's branch 
refs/heads/main from Shane Ardell
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=17084a1f09 ]

NIFI-10538: UI updates for making RegistryClient an extension point (#6437)

* init commit

* remove test string

* display only class name for types combo

* remove commented code

* add UI support for dynamic properties

* add support for controller services

* address review feedback:

* fix syntax error
* remove console logs
* move contents of editRegistry into callback
* add name and description fields back to edit dialog

* add referencing component logic for registry clients

* implement reload logic

* more review findings
* increase size of description textarea in config
* remove location column from registry table
* remove commented out code

* post-rebase tweaks
* change if condition from assignment to evaluation
* add missing semicolons
* refactor new registry initialization into function

* more feedback addressed
* fix rebase error around controller service reference
* render registry references in their own list
* remove duplicate updateValidationError function
* remove TODO comment

* expose reload through nfSettings public variable

* show add registry dialog only after available types returns

* fix css class name

This closes #6437 

> UI changes to support RegsitryClient as an extension point
> --
>
> Key: NIFI-10538
> URL: https://issues.apache.org/jira/browse/NIFI-10538
> Project: Apache NiFi
>  Issue Type: Sub-task
>Reporter: Shane Ardell
>Assignee: Shane Ardell
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (NIFI-10463) Fix GetHubSpot incremental loading

[Peter Turcsanyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Turcsanyi updated NIFI-10463:
---
Fix Version/s: 1.18.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Fix GetHubSpot incremental loading
> --
>
> Key: NIFI-10463
> URL: https://issues.apache.org/jira/browse/NIFI-10463
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Lehel Boér
>Assignee: Lehel Boér
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> GetHubSpot current incremental loading implementation is based on paging and 
> doesn't work correctly. Needs to be updated with a timestamp based approach.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10463) Fix GetHubSpot incremental loading

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609669#comment-17609669
 ] 

ASF subversion and git services commented on NIFI-10463:


Commit 5d2bd1e47b34b39eda4068dbe6f941d525084dbc in nifi's branch 
refs/heads/main from Lehel
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5d2bd1e47b ]

NIFI-10463: Fix GetHubSpot incremental loading

This closes #6379.

Signed-off-by: Peter Turcsanyi 


> Fix GetHubSpot incremental loading
> --
>
> Key: NIFI-10463
> URL: https://issues.apache.org/jira/browse/NIFI-10463
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Lehel Boér
>Assignee: Lehel Boér
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> GetHubSpot current incremental loading implementation is based on paging and 
> doesn't work correctly. Needs to be updated with a timestamp based approach.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] asfgit closed pull request #6379: NIFI-10463: Fix GetHubSpot incremental loading

[GitBox]

asfgit closed pull request #6379: NIFI-10463: Fix GetHubSpot incremental loading
URL: https://github.com/apache/nifi/pull/6379


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] simonbence opened a new pull request, #6451: NIFI-10550 Fixing SSH Context Service validation for NifiRegistryFlowRegistryClient

[GitBox]

simonbence opened a new pull request, #6451:
URL: https://github.com/apache/nifi/pull/6451

   
   
   [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [NIFI-10550](https://issues.apache.org/jira/browse/NIFI-10550)
   
   ### Pull Request Tracking
   
   - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [ ] Pull Request based on current revision of the `main` branch
   - [ ] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [ ] JDK 8
 - [ ] JDK 11
 - [ ] JDK 17
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (NIFI-10550) Fixing SSL context service validation for NifiRegistryFlowRegistryClient

[Simon Bence (Jira)]

Simon Bence created NIFI-10550:
--

 Summary: Fixing SSL context service validation for 
NifiRegistryFlowRegistryClient
 Key: NIFI-10550
 URL: https://issues.apache.org/jira/browse/NIFI-10550
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Simon Bence
Assignee: Simon Bence


The validation of the SSL Context Service within NifiRegistryFlowRegistryClient 
should trigger when the Controller Service is set but only the Trust of Key 
store is filled not both. The current logit triggers a validation error when 
both are set instead, this should be fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (NIFI-10532) PutFTP does not group the batch by user name

[Alex Sherwin (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609662#comment-17609662
 ] 

Alex Sherwin edited comment on NIFI-10532 at 9/26/22 7:11 PM:
--

FWIW I just ran up against this too in PutSFTP.

We have a pattern where we route to hundreds of different destination users 
based on an attribute on the FlowFile for the target username, but all to the 
same target SFTP server.

We pull authentication info using a ControllerService for SSH key based auth 
information (so no sensitive info is in FlowFile attributes).

I traced it down to the batching behavior and just found this ticket, so, make 
that two people that were surprised the optimistic SFTPClient reuse cache is by 
host only, it would make sense to me to be host + port + username (seems 
unlikely the credentials should be accounted for here, the chance of the 
credentials differing for the same host + port + username seems slim)


was (Author: alex.sher...@gmail.com):
FWIW I just ran up against this too in PutSFTP.

We have a pattern where we route to hundreds of different destination users 
based on an attribute on the FlowFile for the target username, but all to the 
same target SFTP server.

We pull authentication info using a ControllerService for SSH key based auth 
information (so no sensitive info is in FlowFile attributes).

I traced it down to the batching behavior and just found this ticket, so, make 
that two people that were surprised the optimistic SFTPClient reuse cache is by 
host only, it would make sense to me to be host + port + username (seems 
unlikely the credentials should be accounted for here, the change of the 
credentials differing for the same host + port + username seems slim)

> PutFTP does not group the batch by user name
> 
>
> Key: NIFI-10532
> URL: https://issues.apache.org/jira/browse/NIFI-10532
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.17.0, 1.16.1
>Reporter: Christoph Langheld
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 1.18.0
>
> Attachments: 01-PutFTP-ProcessGroup.png, 02-PutFTP-Setting.png, 
> 03-PutFTP-Result.png, PutFTP-Bug.xml
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Hello,
> for the PutFTP processor we set the host, name, password, port, and target 
> directory dynamically via UpdateAttribute.
> We now have the problem, that the PutFTP processor transmits every file with 
> the same user even the ftp user name changed. The target host does not 
> change, only the user.
> To reproduce I attached a process group as template ([^PutFTP-Bug.xml]). You 
> have to adapt the ftp server settings within the UpdateAttribute processor to 
> your environment.
> The process group generates 50 flow files.
> UpdateAttribute sets the ftp user credentials and sets a filename prefix 
> (NIFI-A_ respectively NIFI-B_).
> If everything would work correctly all files with prefix NIFI-A_ should be 
> transfered to the ftp server as user nifi-a and the rest as user nifi-b.
> But every file will be transferred as the same user (nifi-a).
> PutFTP should group the batch by host, login credentials (user, password, 
> port) and target directory.
> !01-PutFTP-ProcessGroup.png|width=639,height=607!
> PutFTP settings:
> !02-PutFTP-Setting.png|width=750,height=523!
> Result on the ftp server (logged in as user nifi-a):
> !03-PutFTP-Result.png|width=268,height=482!
>  
> Thank you and regards
> Christoph
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] mr1716 commented on pull request #6449: NIFI-10548 Update Jetty From 9.4.48.v20220622 to 9.4.49.v20220914

[GitBox]

mr1716 commented on PR #6449:
URL: https://github.com/apache/nifi/pull/6449#issuecomment-1258486462

   It took 2 runs of the checks for the checks to run successfully in the pull 
in the branch I created. It might require re-running the failed/cancelled jobs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (NIFI-10532) PutFTP does not group the batch by user name

[Joe Witt (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609663#comment-17609663
 ] 

Joe Witt commented on NIFI-10532:
-

yep this pr fixes this. But doesnt change how much I greatly dislike this 
feature :) 

It could never be efficient generally speaking.  

> PutFTP does not group the batch by user name
> 
>
> Key: NIFI-10532
> URL: https://issues.apache.org/jira/browse/NIFI-10532
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.17.0, 1.16.1
>Reporter: Christoph Langheld
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 1.18.0
>
> Attachments: 01-PutFTP-ProcessGroup.png, 02-PutFTP-Setting.png, 
> 03-PutFTP-Result.png, PutFTP-Bug.xml
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Hello,
> for the PutFTP processor we set the host, name, password, port, and target 
> directory dynamically via UpdateAttribute.
> We now have the problem, that the PutFTP processor transmits every file with 
> the same user even the ftp user name changed. The target host does not 
> change, only the user.
> To reproduce I attached a process group as template ([^PutFTP-Bug.xml]). You 
> have to adapt the ftp server settings within the UpdateAttribute processor to 
> your environment.
> The process group generates 50 flow files.
> UpdateAttribute sets the ftp user credentials and sets a filename prefix 
> (NIFI-A_ respectively NIFI-B_).
> If everything would work correctly all files with prefix NIFI-A_ should be 
> transfered to the ftp server as user nifi-a and the rest as user nifi-b.
> But every file will be transferred as the same user (nifi-a).
> PutFTP should group the batch by host, login credentials (user, password, 
> port) and target directory.
> !01-PutFTP-ProcessGroup.png|width=639,height=607!
> PutFTP settings:
> !02-PutFTP-Setting.png|width=750,height=523!
> Result on the ftp server (logged in as user nifi-a):
> !03-PutFTP-Result.png|width=268,height=482!
>  
> Thank you and regards
> Christoph
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10532) PutFTP does not group the batch by user name

[Alex Sherwin (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609662#comment-17609662
 ] 

Alex Sherwin commented on NIFI-10532:
-

FWIW I just ran up against this too in PutSFTP.

We have a pattern where we route to hundreds of different destination users 
based on an attribute on the FlowFile for the target username, but all to the 
same target SFTP server.

We pull authentication info using a ControllerService for SSH key based auth 
information (so no sensitive info is in FlowFile attributes).

I traced it down to the batching behavior and just found this ticket, so, make 
that two people that were surprised the optimistic SFTPClient reuse cache is by 
host only, it would make sense to me to be host + port + username (seems 
unlikely the credentials should be accounted for here, the change of the 
credentials differing for the same host + port + username seems slim)

> PutFTP does not group the batch by user name
> 
>
> Key: NIFI-10532
> URL: https://issues.apache.org/jira/browse/NIFI-10532
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.17.0, 1.16.1
>Reporter: Christoph Langheld
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 1.18.0
>
> Attachments: 01-PutFTP-ProcessGroup.png, 02-PutFTP-Setting.png, 
> 03-PutFTP-Result.png, PutFTP-Bug.xml
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Hello,
> for the PutFTP processor we set the host, name, password, port, and target 
> directory dynamically via UpdateAttribute.
> We now have the problem, that the PutFTP processor transmits every file with 
> the same user even the ftp user name changed. The target host does not 
> change, only the user.
> To reproduce I attached a process group as template ([^PutFTP-Bug.xml]). You 
> have to adapt the ftp server settings within the UpdateAttribute processor to 
> your environment.
> The process group generates 50 flow files.
> UpdateAttribute sets the ftp user credentials and sets a filename prefix 
> (NIFI-A_ respectively NIFI-B_).
> If everything would work correctly all files with prefix NIFI-A_ should be 
> transfered to the ftp server as user nifi-a and the rest as user nifi-b.
> But every file will be transferred as the same user (nifi-a).
> PutFTP should group the batch by host, login credentials (user, password, 
> port) and target directory.
> !01-PutFTP-ProcessGroup.png|width=639,height=607!
> PutFTP settings:
> !02-PutFTP-Setting.png|width=750,height=523!
> Result on the ftp server (logged in as user nifi-a):
> !03-PutFTP-Result.png|width=268,height=482!
>  
> Thank you and regards
> Christoph
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10460) Zendesk Support source connector

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609657#comment-17609657
 ] 

ASF subversion and git services commented on NIFI-10460:


Commit 0fa923159f0d238e986a8dbe6edfc2acbe785f68 in nifi's branch 
refs/heads/main from Ferenc Kis
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=0fa923159f ]

NIFI-10460 GetZendesk processor

Signed-off-by: Arpad Boda 

This closes #6411


> Zendesk Support source connector
> 
>
> Key: NIFI-10460
> URL: https://issues.apache.org/jira/browse/NIFI-10460
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Ferenc Kis
>Assignee: Ferenc Kis
>Priority: Major
>  Labels: Zendesk
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Connector to incrementally fetch data from Zendesk Support.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] arpadboda closed pull request #6411: NIFI-10460 GetZendesk processor

[GitBox]

arpadboda closed pull request #6411: NIFI-10460 GetZendesk processor
URL: https://github.com/apache/nifi/pull/6411


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] tpalfy commented on a diff in pull request #6444: NIFI-10513: Added capture non-record fields to JsonTreeRowRecordReader

[GitBox]

tpalfy commented on code in PR #6444:
URL: https://github.com/apache/nifi/pull/6444#discussion_r980339020


##
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-json-record-utils/src/main/java/org/apache/nifi/json/AbstractJsonRowRecordReader.java:
##
@@ -76,26 +81,60 @@ private AbstractJsonRowRecordReader(final ComponentLog 
logger, final String date
 LAZY_TIMESTAMP_FORMAT = () -> tsf;
 }
 
-protected AbstractJsonRowRecordReader(final InputStream in, final 
ComponentLog logger, final String dateFormat, final String timeFormat, final 
String timestampFormat)
+protected AbstractJsonRowRecordReader(final InputStream in,
+  final ComponentLog logger,
+  final String dateFormat,
+  final String timeFormat,
+  final String timestampFormat)
 throws IOException, MalformedRecordException {
 
-this(in, logger, dateFormat, timeFormat, timestampFormat, null, null);
+this(in, logger, dateFormat, timeFormat, timestampFormat, null, null, 
null);
 }
 
-protected AbstractJsonRowRecordReader(final InputStream in, final 
ComponentLog logger, final String dateFormat, final String timeFormat, final 
String timestampFormat,
-  final StartingFieldStrategy 
strategy, final String nestedFieldName) throws IOException, 
MalformedRecordException {
+/**
+ * Constructor with initial logic for JSON to NiFi record parsing.
+ *
+ * @param in the input stream to parse
+ * @param logger ComponentLog
+ * @param dateFormat format for parsing date fields
+ * @param timeFormat format for parsing time fields
+ * @param timestampFormatformat for parsing timestamp fields
+ * @param strategy   whether to start processing from a 
specific field
+ * @param nestedFieldNamethe name of the field to start the 
processing from
+ * @param captureFieldPredicate predicate that takes a JSON fieldName and 
fieldValue to capture top-level non-processed fields which can
+ *   be accessed by calling {@link 
#getCapturedFields()}
+ * @throws IOException  in case of JSON stream processing 
failure
+ * @throws MalformedRecordException in case of malformed JSON input
+ */
+protected AbstractJsonRowRecordReader(final InputStream in,
+  final ComponentLog logger,
+  final String dateFormat,
+  final String timeFormat,
+  final String timestampFormat,
+  final StartingFieldStrategy strategy,
+  final String nestedFieldName,
+  final BiPredicate 
captureFieldPredicate)
+throws IOException, MalformedRecordException {
 
 this(logger, dateFormat, timeFormat, timestampFormat);
 
 this.strategy = strategy;
+this.captureFieldPredicate = captureFieldPredicate;
+capturedFields = new HashMap<>();
 
 try {
 jsonParser = jsonFactory.createParser(in);
 jsonParser.setCodec(codec);
 
 if (strategy == StartingFieldStrategy.NESTED_FIELD) {
-final SerializedString serializedStartingFieldName = new 
SerializedString(nestedFieldName);
-while (!jsonParser.nextFieldName(serializedStartingFieldName) 
&& jsonParser.hasCurrentToken());
+while (jsonParser.nextToken() != null) {
+if (nestedFieldName.equals(jsonParser.getCurrentName())) {
+break;
+}

Review Comment:
   ```suggestion
   if (nestedFieldName.equals(jsonParser.getCurrentName())) 
{
   logger.debug("Parsing starting at nested field 
[{}]", nestedFieldName);
   break;
   }
   ```



##
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-json-record-utils/src/main/java/org/apache/nifi/json/AbstractJsonRowRecordReader.java:
##
@@ -76,26 +81,60 @@ private AbstractJsonRowRecordReader(final ComponentLog 
logger, final String date
 LAZY_TIMESTAMP_FORMAT = () -> tsf;
 }
 
-protected AbstractJsonRowRecordReader(final InputStream in, final 
ComponentLog logger, final String dateFormat, final String timeFormat, final 
String timestampFormat)
+protected AbstractJsonRowRecordReader(final InputStream in,
+  final ComponentLog logger,
+  final String dateFormat,
+

[GitHub] [nifi] sashashura opened a new pull request, #6450: GitHub Workflows security hardening

[GitBox]

sashashura opened a new pull request, #6450:
URL: https://github.com/apache/nifi/pull/6450

   This PR adds explicit [permissions 
section](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions)
 to workflows. This is a security best practice because by default workflows 
run with [extended set of 
permissions](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token)
 (except from `on: pull_request` [from external 
forks](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/)).
 By specifying any permission explicitly all others are set to none. By using 
the principle of least privilege the damage a compromised workflow can do 
(because of an 
[injection](https://securitylab.github.com/research/github-actions-untrusted-input/)
 or compromised third party tool or action) is restricted.
   It is recommended to have [most strict permissions on the top 
level](https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions)
 and grant write permissions on [job 
level](https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs)
 case by case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-10549) NiFi doesn't start if include-ranger profile is specified

[Bryan Bende (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende updated NIFI-10549:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> NiFi doesn't start if include-ranger profile is specified
> -
>
> Key: NIFI-10549
> URL: https://issues.apache.org/jira/browse/NIFI-10549
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> If the `-Pinclude-ranger` profile is included in the Maven build command for 
> NiFi, nifi-ranger-resources.jar and all its dependencies are put into lib, 
> causing the following error on startup:
> Failure to launch NiFi java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/JsonNode at 
> org.apache.nifi.util.hive.HiveJdbcCommon.(HiveJdbcCommon.java:92) at 
> org.apache.nifi.processors.hive.SelectHive3QL.(SelectHive3QL.java:272)
>  at java.lang.Class.forName0(Native Method) at 
> java.lang.Class.forName(Class.java:348) at 
> org.apache.nifi.nar.StandardExtensionDiscoveringManager.getClass(StandardExtensionDiscoveringManager.java:338)
>  at 
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:102)
>  at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:66) 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:795) at 
> org.apache.nifi.NiFi.(NiFi.java:172) at 
> org.apache.nifi.NiFi.(NiFi.java:83) at 
> org.apache.nifi.NiFi.main(NiFi.java:332) Caused by: 
> java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.JsonNode at 
> java.net.URLClassLoader.findClass(URLClassLoader.java:387) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:418) at 
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 11 common frames 
> omitted
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10549) NiFi doesn't start if include-ranger profile is specified

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609604#comment-17609604
 ] 

ASF subversion and git services commented on NIFI-10549:


Commit 363e7981079dd13eab9f15be48b3de15657db6fe in nifi's branch 
refs/heads/main from Matt Burgess
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=363e798107 ]

NIFI-10549: Remove group name wildcard from assembly for nifi-ranger-resources 
(#6448)



> NiFi doesn't start if include-ranger profile is specified
> -
>
> Key: NIFI-10549
> URL: https://issues.apache.org/jira/browse/NIFI-10549
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> If the `-Pinclude-ranger` profile is included in the Maven build command for 
> NiFi, nifi-ranger-resources.jar and all its dependencies are put into lib, 
> causing the following error on startup:
> Failure to launch NiFi java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/JsonNode at 
> org.apache.nifi.util.hive.HiveJdbcCommon.(HiveJdbcCommon.java:92) at 
> org.apache.nifi.processors.hive.SelectHive3QL.(SelectHive3QL.java:272)
>  at java.lang.Class.forName0(Native Method) at 
> java.lang.Class.forName(Class.java:348) at 
> org.apache.nifi.nar.StandardExtensionDiscoveringManager.getClass(StandardExtensionDiscoveringManager.java:338)
>  at 
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:102)
>  at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:66) 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:795) at 
> org.apache.nifi.NiFi.(NiFi.java:172) at 
> org.apache.nifi.NiFi.(NiFi.java:83) at 
> org.apache.nifi.NiFi.main(NiFi.java:332) Caused by: 
> java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.JsonNode at 
> java.net.URLClassLoader.findClass(URLClassLoader.java:387) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:418) at 
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 11 common frames 
> omitted
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] bbende merged pull request #6448: NIFI-10549: Remove group name wildcard from assembly for nifi-ranger-resources

[GitBox]

bbende merged PR #6448:
URL: https://github.com/apache/nifi/pull/6448


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] bbende commented on pull request #6448: NIFI-10549: Remove group name wildcard from assembly for nifi-ranger-resources

[GitBox]

bbende commented on PR #6448:
URL: https://github.com/apache/nifi/pull/6448#issuecomment-1258372395

   Looks good, will merge


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Assigned] (NIFI-10548) Update Jetty From 9.4.48.v20220622 to 9.4.49.v20220914

[Mike R (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike R reassigned NIFI-10548:
-

Assignee: Mike R

> Update Jetty From 9.4.48.v20220622 to 9.4.49.v20220914
> --
>
> Key: NIFI-10548
> URL: https://issues.apache.org/jira/browse/NIFI-10548
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Mike R
>Assignee: Mike R
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Update Jetty From 9.4.48.v20220622 to 9.4.49.v20220914



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] mr1716 opened a new pull request, #6449: NIFI-10548 Update Jetty From 9.4.48.v20220622 to 9.4.49.v20220914

[GitBox]

mr1716 opened a new pull request, #6449:
URL: https://github.com/apache/nifi/pull/6449

   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   
   [NIFI-10548](https://issues.apache.org/jira/browse/NIFI-10548)  Update Jetty 
From 9.4.48.v20220622 to 9.4.49.v20220914
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [X] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [X] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [X] Pull Request based on current revision of the `main` branch
   - [X] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [X] JDK 8
 - [X] JDK 11
 - [X] JDK 17
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-10497) Making RegistryClient an extension point

[Mark Payne (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-10497:
--
Fix Version/s: 1.18.0

> Making RegistryClient an extension point
> 
>
> Key: NIFI-10497
> URL: https://issues.apache.org/jira/browse/NIFI-10497
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Critical
> Fix For: 1.18.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Currently NiFi is capable to connect only to the NiFi Registry as a "registry 
> repository". It would be beneficial to give NiFi the capability to depend on 
> other services.
> In order to this, I suggest to decouple the registry behaviour from the NiFi 
> Registry as much as it is possible (inlcuding but not only the API and the 
> Resources) and move the actual implementation behind this new API.
> To be able to add new implementations, this new API must be an extension 
> point applying the usual NiFi instruments for this. Also it is paramunt to 
> keep the continuity with the current usages and make the implementation 
> capable to process the current REST call format. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10497) Making RegistryClient an extension point

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609574#comment-17609574
 ] 

ASF subversion and git services commented on NIFI-10497:


Commit 94fdc794b09317ebbcb1edf89dd29d286b1b4554 in nifi's branch 
refs/heads/main from simonbence
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=94fdc794b0 ]

NIFI-10497 Making RegistryClient an extension point (#6433)

- NIFI-10497 Mae FlowRegistryClient an extension point

> Making RegistryClient an extension point
> 
>
> Key: NIFI-10497
> URL: https://issues.apache.org/jira/browse/NIFI-10497
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Critical
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Currently NiFi is capable to connect only to the NiFi Registry as a "registry 
> repository". It would be beneficial to give NiFi the capability to depend on 
> other services.
> In order to this, I suggest to decouple the registry behaviour from the NiFi 
> Registry as much as it is possible (inlcuding but not only the API and the 
> Resources) and move the actual implementation behind this new API.
> To be able to add new implementations, this new API must be an extension 
> point applying the usual NiFi instruments for this. Also it is paramunt to 
> keep the continuity with the current usages and make the implementation 
> capable to process the current REST call format. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] markap14 merged pull request #6433: NIFI-10497 Making RegistryClient an extension point

[GitBox]

markap14 merged PR #6433:
URL: https://github.com/apache/nifi/pull/6433


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] markap14 commented on pull request #6433: NIFI-10497 Making RegistryClient an extension point

[GitBox]

markap14 commented on PR #6433:
URL: https://github.com/apache/nifi/pull/6433#issuecomment-1258290052

   Thanks for all the work you've done here @simonbence! This is a huge 
changeset and really great work put together here. I did quite a lot of testing:
   - Basics of adding a registry, starting version control, stopping version 
control, changing version, view local modifications, revert changes.
   - Built flow that use registry using 1.16, then moved that flow into a nifi 
built on this branch, in order to simulate an upgrade form an older version of 
NiFi
   - Built flow that uses embedded versioned flows
   - Built flow that uses embedded versioned flow in 1.16 and copied over to 
instance from this branch
   - Tried to remove a registry client that was in use, which correctly failed
   - Verified dirty / up-to-date / conflict / new version available states
   - Tested standalone and clustered
   - Tested flows with embedded PG's that are not themselves versioned
   
   All seems to check out with the current build.
   +1 will merge to main.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1412: MINIFICPP-1923 Refactor PutUDP to use asio

[GitBox]

fgerlits commented on code in PR #1412:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1412#discussion_r980221936


##
extensions/standard-processors/tests/unit/ListenSyslogTests.cpp:
##
@@ -269,19 +258,37 @@ TEST_CASE("ListenSyslog without parsing test", 
"[ListenSyslog]") {
   std::string protocol;
 
   SECTION("UDP") {
+asio::ip::udp::endpoint endpoint;
+SECTION("sending through IPv4", "[IPv4]") {
+  endpoint = asio::ip::udp::endpoint(asio::ip::address_v4::loopback(), 
SYSLOG_PORT);
+}
+SECTION("sending through IPv6", "[IPv6]") {
+  if (utils::isIPv6Disabled())
+return;

Review Comment:
   this will exit the test case, so the "TCP" section won't be run
   
   EDIT: no, I'm wrong, this is fine.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1412: MINIFICPP-1923 Refactor PutUDP to use asio

[GitBox]

fgerlits commented on code in PR #1412:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1412#discussion_r980240978


##
libminifi/test/Utils.h:
##
@@ -111,24 +111,42 @@ bool countLogOccurrencesUntil(const std::string& pattern,
   return false;
 }
 
-bool sendMessagesViaTCP(const std::vector& contents, 
uint64_t port) {
+bool sendMessagesViaTCP(const std::vector& contents, const 
asio::ip::tcp::endpoint& remote_endpoint) {
   asio::io_context io_context;
   asio::ip::tcp::socket socket(io_context);
-  asio::ip::tcp::endpoint 
remote_endpoint(asio::ip::address::from_string("127.0.0.1"), port);
   socket.connect(remote_endpoint);
   std::error_code err;
   for (auto& content : contents) {
 std::string tcp_message(content);
 tcp_message += '\n';
 asio::write(socket, asio::buffer(tcp_message, tcp_message.size()), err);

Review Comment:
   Won't this overwrite the previous value of `err`?  I.e., if there are 2 
writes, the first fails and the second succeeds, then we'll return `true` as if 
everything was A-OK.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1412: MINIFICPP-1923 Refactor PutUDP to use asio

[GitBox]

fgerlits commented on code in PR #1412:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1412#discussion_r980225624


##
extensions/standard-processors/tests/unit/ListenSyslogTests.cpp:
##
@@ -197,21 +197,10 @@ constexpr std::string_view rfc5424_logger_example_1 = 
R"(<13>1 2022-03-17T10:10:
 
 constexpr std::string_view invalid_syslog = "not syslog";
 
-void sendUDPPacket(const std::string_view content, uint64_t port) {
-  asio::io_context io_context;
-  asio::ip::udp::socket socket(io_context);
-  asio::ip::udp::endpoint 
remote_endpoint(asio::ip::address::from_string("127.0.0.1"), port);
-  socket.open(asio::ip::udp::v4());
-  std::error_code err;
-  socket.send_to(asio::buffer(content, content.size()), remote_endpoint, 0, 
err);
-  REQUIRE(!err);
-  socket.close();
-}
-
 void check_for_only_basic_attributes(core::FlowFile& flow_file, uint16_t port, 
std::string_view protocol) {
   CHECK(std::to_string(port) == flow_file.getAttribute("syslog.port"));
   CHECK(protocol == flow_file.getAttribute("syslog.protocol"));
-  CHECK("127.0.0.1" == flow_file.getAttribute("syslog.sender"));
+  CHECK((":::127.0.0.1" == flow_file.getAttribute("syslog.sender") || 
"::1" == flow_file.getAttribute("syslog.sender")));

Review Comment:
   this seems to be only IPv6 now; how does this work in the IPv4 case?



##
extensions/standard-processors/processors/PutUDP.cpp:
##
@@ -107,51 +98,53 @@ void PutUDP::onTrigger(core::ProcessContext* context, 
core::ProcessSession* cons
 return;
   }
 
-  const auto nonthrowing_sockaddr_ntop = [](const sockaddr* const sa) -> 
std::string {
-return utils::try_expression([sa] { return utils::net::sockaddr_ntop(sa); 
}).value_or("(n/a)");
+  asio::io_context io_context;
+
+  const auto resolve_hostname = [&io_context, &hostname, &port]() -> 
nonstd::expected {
+udp::resolver resolver(io_context);
+std::error_code error_code;
+auto resolved_query = resolver.resolve(hostname, port, error_code);
+if (error_code)
+  return nonstd::make_unexpected(error_code);
+return resolved_query;
   };
 
-  const auto debug_log_resolved_names = [&, this](const addrinfo& names) -> 
decltype(auto) {
-if (logger_->should_log(core::logging::LOG_LEVEL::debug)) {
-  std::vector names_vector;
-  for (const addrinfo* it = &names; it; it = it->ai_next) {
-names_vector.push_back(nonthrowing_sockaddr_ntop(it->ai_addr));
+  const auto send_data_to_endpoint = [&io_context, &data, &logger = 
this->logger_](const udp::resolver::results_type& resolved_query) -> 
nonstd::expected {
+std::error_code error;
+for (const auto& resolver_entry : resolved_query) {
+  error.clear();
+  udp::socket socket(io_context);
+  socket.open(resolver_entry.endpoint().protocol(), error);
+  if (error) {
+logger->log_debug("opening %s socket failed due to %s ", 
resolver_entry.endpoint().protocol() == udp::v4() ? "IPv4" : "IPv6", 
error.message());
+continue;
   }
-  logger_->log_debug("resolved \'%s\' to: %s",
-  hostname,
-  names_vector | ranges::views::join(',') | ranges::to());
+  socket.send_to(asio::buffer(data.buffer), resolver_entry.endpoint(), 
udp::socket::message_flags{}, error);
+  if (error) {
+core::logging::LOG_DEBUG(logger) << "sending to endpoint " << 
resolver_entry.endpoint() << " failed due to " << error.message();

Review Comment:
   Are these errors expected and harmless?  If not, we should log it on error 
(or warning) level; also at line 119.
   
   Also, I think it would be useful to add a debug-level log in the successful 
case, so we can see that "sending to X failed", "sending to Y failed", "sending 
to Z succeeded".



##
libminifi/test/Utils.h:
##
@@ -111,24 +111,42 @@ bool countLogOccurrencesUntil(const std::string& pattern,
   return false;
 }
 
-bool sendMessagesViaTCP(const std::vector& contents, 
uint64_t port) {
+bool sendMessagesViaTCP(const std::vector& contents, const 
asio::ip::tcp::endpoint& remote_endpoint) {
   asio::io_context io_context;
   asio::ip::tcp::socket socket(io_context);
-  asio::ip::tcp::endpoint 
remote_endpoint(asio::ip::address::from_string("127.0.0.1"), port);
   socket.connect(remote_endpoint);
   std::error_code err;
   for (auto& content : contents) {
 std::string tcp_message(content);
 tcp_message += '\n';
 asio::write(socket, asio::buffer(tcp_message, tcp_message.size()), err);
   }
-  if (err) {
+  if (err)
+return false;
+  socket.close();
+  return true;

Review Comment:
   I can see this was like this before, but why don't we (try to) close the 
socket in the error case?  Maybe the `connect()` and the first `write()` were 
successful, only (say) the second `write()` failed.



##
extensions/standard-processors/tests/unit/ListenSyslogTests.cpp:
##
@@ -269,19 +258,37 @@ TEST_CASE("ListenSyslog without parsing test", 
"[ListenSyslog]") {
   std::string protocol;

[GitHub] [nifi] tamas-horvath commented on pull request #6448: NIFI-10549: Remove group name wildcard from assembly for nifi-ranger-resources

[GitBox]

tamas-horvath commented on PR #6448:
URL: https://github.com/apache/nifi/pull/6448#issuecomment-1258229271

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

fgerlits commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r980174559


##
libminifi/include/core/json/JsonConfiguration.h:
##
@@ -0,0 +1,71 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+
+#include "core/FlowConfiguration.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/ProcessorConfig.h"
+#include "Exception.h"
+#include "io/StreamFactory.h"
+#include "io/validation.h"
+#include "sitetosite/SiteToSite.h"
+#include "utils/Id.h"
+#include "utils/StringUtils.h"
+#include "utils/file/FileSystem.h"
+#include "core/flow/StructuredConfiguration.h"
+
+class JsonConfigurationTestAccessor;
+
+namespace org::apache::nifi::minifi::core {
+
+class JsonConfiguration : public StructuredConfiguration {
+ public:
+  explicit JsonConfiguration(ConfigurationContext ctx);
+
+  ~JsonConfiguration() override = default;
+
+  /**
+   * Returns a shared pointer to a ProcessGroup object containing the

Review Comment:
   I would have removed the whole comment blocks, as they look a bit weird now. 
 But if you think the remaining parts of the comments are useful, I don't mind 
keeping them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] bejancsaba commented on pull request #6411: NIFI-10460 GetZendesk processor

[GitBox]

bejancsaba commented on PR #6411:
URL: https://github.com/apache/nifi/pull/6411#issuecomment-1258195567

   > I'm good and happy to merge as soon as @bejancsaba is happy with this, too.
   
   @arpadboda this PR is good from my side. I'm not sure whether anyone with 
more NiFi expertise will take a look or not.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] bejancsaba commented on a diff in pull request #6411: NIFI-10460 GetZendesk processor

[GitBox]

bejancsaba commented on code in PR #6411:
URL: https://github.com/apache/nifi/pull/6411#discussion_r980168036


##
nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-processors/src/test/java/org/apache/nifi/processors/zendesk/GetZendeskTest.java:
##
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.zendesk;
+
+import static java.lang.String.format;
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.HTTP_TOO_MANY_REQUESTS;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.RECORD_COUNT_ATTRIBUTE_NAME;
+import static org.apache.nifi.processors.zendesk.GetZendesk.REL_SUCCESS_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.WEB_CLIENT_SERVICE_PROVIDER_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_EXPORT_METHOD_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_PASSWORD_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_QUERY_START_TIMESTAMP_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_RESOURCE_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_SUBDOMAIN_NAME;
+import static org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_TOKEN_NAME;
+import static org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_USER_NAME;
+import static org.apache.nifi.processors.zendesk.ZendeskExportMethod.CURSOR;
+import static org.apache.nifi.processors.zendesk.ZendeskResource.TICKETS;
+import static org.apache.nifi.util.TestRunners.newTestRunner;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Stream;
+import okhttp3.HttpUrl;
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import okhttp3.mockwebserver.RecordedRequest;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.web.client.StandardHttpUriBuilder;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import 
org.apache.nifi.web.client.provider.service.StandardWebClientServiceProvider;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.opentest4j.AssertionFailedError;
+
+public class GetZendeskTest {
+
+private static final int HTTP_OK = 200;
+private static final int HTTP_BAD_REQUEST = 400;
+
+private static final String DEFAULT_QUERY_START_TIMESTAMP = "1640995200";
+private static final String DEFAULT_CURSOR_VALUE = "123456789";
+private static final String EMPTY_RESPONSE = "{}";
+private static final String RESPONSE_WITH_CURSOR_FIELD_TEMPLATE = "{ 
\"%s\": " + DEFAULT_CURSOR_VALUE + " }";
+private static final String THREE_RECORDS = 
"[{\"id\":1},{\"id\":2},{\"id\":3}]";
+private static final String RESPONSE_WITH_THREE_RECORDS_TEMPLATE = "{ 
\"%s\": " + THREE_RECORDS + " }";
+private static final String RESPONSE_WITH_ZERO_RECORDS_TEMPLATE = "{ 
\"%s\": [] }";
+
+private MockWebServer server;
+private TestRunner testRunner;
+
+private static Stream 
supportedZendeskResourcesExportMethodCombinations() {
+return Stream.of(ZendeskResource.values())
+.flatMap(zendeskResource -> 
zendeskResource.getSupportedExportMethods()
+.stream()
+.map(exportMethod -> Arguments.of(zendeskResource, 
exportMethod)));
+}
+
+private static Stream 
unsupportedZendeskResourcesExpor

[GitHub] [nifi] bejancsaba commented on a diff in pull request #6411: NIFI-10460 GetZendesk processor

[GitBox]

bejancsaba commented on code in PR #6411:
URL: https://github.com/apache/nifi/pull/6411#discussion_r980167345


##
nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-processors/src/test/java/org/apache/nifi/processors/zendesk/GetZendeskTest.java:
##
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.zendesk;
+
+import static java.lang.String.format;
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonMap;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.HTTP_TOO_MANY_REQUESTS;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.RECORD_COUNT_ATTRIBUTE_NAME;
+import static org.apache.nifi.processors.zendesk.GetZendesk.REL_SUCCESS_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.WEB_CLIENT_SERVICE_PROVIDER_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_EXPORT_METHOD_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_PASSWORD_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_QUERY_START_TIMESTAMP_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_RESOURCE_NAME;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_SUBDOMAIN_NAME;
+import static org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_TOKEN_NAME;
+import static org.apache.nifi.processors.zendesk.GetZendesk.ZENDESK_USER_NAME;
+import static org.apache.nifi.processors.zendesk.ZendeskExportMethod.CURSOR;
+import static org.apache.nifi.processors.zendesk.ZendeskResource.TICKETS;
+import static org.apache.nifi.util.TestRunners.newTestRunner;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Stream;
+import okhttp3.HttpUrl;
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import okhttp3.mockwebserver.RecordedRequest;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.web.client.StandardHttpUriBuilder;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import 
org.apache.nifi.web.client.provider.service.StandardWebClientServiceProvider;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.opentest4j.AssertionFailedError;
+
+public class GetZendeskTest {
+
+private static final int HTTP_OK = 200;
+private static final int HTTP_BAD_REQUEST = 400;
+
+private static final String DEFAULT_QUERY_START_TIMESTAMP = "1640995200";
+private static final String DEFAULT_CURSOR_VALUE = "123456789";
+private static final String EMPTY_RESPONSE = "{}";
+private static final String RESPONSE_WITH_CURSOR_FIELD_TEMPLATE = "{ 
\"%s\": " + DEFAULT_CURSOR_VALUE + " }";
+private static final String THREE_RECORDS = 
"[{\"id\":1},{\"id\":2},{\"id\":3}]";
+private static final String RESPONSE_WITH_THREE_RECORDS_TEMPLATE = "{ 
\"%s\": " + THREE_RECORDS + " }";
+private static final String RESPONSE_WITH_ZERO_RECORDS_TEMPLATE = "{ 
\"%s\": [] }";
+
+private MockWebServer server;
+private TestRunner testRunner;
+
+private static Stream 
supportedZendeskResourcesExportMethodCombinations() {
+return Stream.of(ZendeskResource.values())
+.flatMap(zendeskResource -> 
zendeskResource.getSupportedExportMethods()
+.stream()
+.map(exportMethod -> Arguments.of(zendeskResource, 
exportMethod)));
+}
+
+private static Stream 
unsupportedZendeskResourcesExpor

[jira] [Updated] (NIFI-10549) NiFi doesn't start if include-ranger profile is specified

[Matt Burgess (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-10549:

Status: Patch Available  (was: In Progress)

> NiFi doesn't start if include-ranger profile is specified
> -
>
> Key: NIFI-10549
> URL: https://issues.apache.org/jira/browse/NIFI-10549
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.18.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> If the `-Pinclude-ranger` profile is included in the Maven build command for 
> NiFi, nifi-ranger-resources.jar and all its dependencies are put into lib, 
> causing the following error on startup:
> Failure to launch NiFi java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/JsonNode at 
> org.apache.nifi.util.hive.HiveJdbcCommon.(HiveJdbcCommon.java:92) at 
> org.apache.nifi.processors.hive.SelectHive3QL.(SelectHive3QL.java:272)
>  at java.lang.Class.forName0(Native Method) at 
> java.lang.Class.forName(Class.java:348) at 
> org.apache.nifi.nar.StandardExtensionDiscoveringManager.getClass(StandardExtensionDiscoveringManager.java:338)
>  at 
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:102)
>  at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:66) 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:795) at 
> org.apache.nifi.NiFi.(NiFi.java:172) at 
> org.apache.nifi.NiFi.(NiFi.java:83) at 
> org.apache.nifi.NiFi.main(NiFi.java:332) Caused by: 
> java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.JsonNode at 
> java.net.URLClassLoader.findClass(URLClassLoader.java:387) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:418) at 
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 11 common frames 
> omitted
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] mattyb149 opened a new pull request, #6448: NIFI-10549: Remove group name wildcard from assembly for nifi-ranger-resources

[GitBox]

mattyb149 opened a new pull request, #6448:
URL: https://github.com/apache/nifi/pull/6448

   # Summary
   
   [NIFI-10549](https://issues.apache.org/jira/browse/NIFI-10549)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [x] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [x] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [x] Pull Request based on current revision of the `main` branch
   - [x] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [ ] Build completed using `mvn clean install -P contrib-check`
 - [ ] JDK 8
 - [x] JDK 11
 - [ ] JDK 17
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

fgerlits commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r980160534


##
libminifi/include/core/flow/StructuredConfiguration.h:
##
@@ -0,0 +1,234 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+
+#include "core/FlowConfiguration.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/ProcessorConfig.h"
+#include "Exception.h"
+#include "io/StreamFactory.h"
+#include "io/validation.h"
+#include "sitetosite/SiteToSite.h"
+#include "utils/Id.h"
+#include "utils/StringUtils.h"
+#include "utils/file/FileSystem.h"
+#include "core/flow/Node.h"
+
+namespace org::apache::nifi::minifi::core {
+
+static constexpr char const* CONFIG_FLOW_CONTROLLER_KEY = "Flow Controller";
+static constexpr char const* CONFIG_PROCESSORS_KEY = "Processors";
+static constexpr char const* CONFIG_CONTROLLER_SERVICES_KEY = "Controller 
Services";
+static constexpr char const* CONFIG_REMOTE_PROCESS_GROUP_KEY = "Remote 
Processing Groups";
+static constexpr char const* CONFIG_REMOTE_PROCESS_GROUP_KEY_V3 = "Remote 
Process Groups";
+static constexpr char const* CONFIG_PROVENANCE_REPORT_KEY = "Provenance 
Reporting";
+static constexpr char const* CONFIG_FUNNELS_KEY = "Funnels";
+
+#define CONFIGURATION_USE_REGEX
+
+// Disable regex in EL for incompatible compilers
+#if __GNUC__ < 4 || (__GNUC__ == 4 && __GNUC_MINOR__ < 9)
+#undef CONFIGURATION_USE_REGEX
+#endif

Review Comment:
   can you remove it from `YamlConfiguration.cpp` and 
`YamlConfigurationTests.cpp`, too, please?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] bejancsaba commented on a diff in pull request #6411: NIFI-10460 GetZendesk processor

[GitBox]

bejancsaba commented on code in PR #6411:
URL: https://github.com/apache/nifi/pull/6411#discussion_r980151120


##
nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-processors/src/main/java/org/apache/nifi/processors/zendesk/GetZendesk.java:
##
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.zendesk;
+
+import static com.fasterxml.jackson.core.JsonEncoding.UTF8;
+import static com.fasterxml.jackson.core.JsonToken.FIELD_NAME;
+import static com.fasterxml.jackson.core.JsonToken.VALUE_NULL;
+import static java.lang.String.format;
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static java.util.Arrays.asList;
+import static java.util.Base64.getEncoder;
+import static java.util.Collections.singleton;
+import static java.util.Collections.singletonMap;
+import static java.util.Collections.unmodifiableList;
+import static 
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_FORBIDDEN;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static 
org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static 
org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static 
org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static 
org.apache.nifi.processor.util.StandardValidators.POSITIVE_LONG_VALIDATOR;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.RECORD_COUNT_ATTRIBUTE_NAME;
+import static org.apache.nifi.util.StringUtils.isNotBlank;
+import static org.apache.nifi.web.client.api.HttpResponseStatus.OK;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@InputRequirement(INPUT_FORBIDDEN)
+@DefaultSettings(yieldDuration = "20 sec")
+@Tags({"zendesk"})
+@CapabilityDescription("Incrementally fetches data from Zendesk API.")
+@Stateful(scopes = CLUSTER, description = "Paging cursor for Zendesk API is 
stored. Cursor is updated after each successful request.")
+@WritesAttributes({
+@WritesAttribute(attribute = RECORD_COUNT_ATTRIBUTE_NAME, description = 
"The number of records fetched by the processor.")})
+public class GetZendesk extends AbstractProcessor {
+
+static final int HTTP_TOO_MANY_REQUESTS = 429;
+static final String RECORD_COUNT_ATTRIBUTE_NAME = "record.count";
+

[GitHub] [nifi] bejancsaba commented on a diff in pull request #6411: NIFI-10460 GetZendesk processor

[GitBox]

bejancsaba commented on code in PR #6411:
URL: https://github.com/apache/nifi/pull/6411#discussion_r980150514


##
nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-processors/src/main/java/org/apache/nifi/processors/zendesk/GetZendesk.java:
##
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.zendesk;
+
+import static com.fasterxml.jackson.core.JsonEncoding.UTF8;
+import static com.fasterxml.jackson.core.JsonToken.FIELD_NAME;
+import static com.fasterxml.jackson.core.JsonToken.VALUE_NULL;
+import static java.lang.String.format;
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static java.util.Arrays.asList;
+import static java.util.Base64.getEncoder;
+import static java.util.Collections.singleton;
+import static java.util.Collections.singletonMap;
+import static java.util.Collections.unmodifiableList;
+import static 
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_FORBIDDEN;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static 
org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static 
org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static 
org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static 
org.apache.nifi.processor.util.StandardValidators.POSITIVE_LONG_VALIDATOR;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.RECORD_COUNT_ATTRIBUTE_NAME;
+import static org.apache.nifi.util.StringUtils.isNotBlank;
+import static org.apache.nifi.web.client.api.HttpResponseStatus.OK;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@InputRequirement(INPUT_FORBIDDEN)
+@DefaultSettings(yieldDuration = "20 sec")
+@Tags({"zendesk"})
+@CapabilityDescription("Incrementally fetches data from Zendesk API.")
+@Stateful(scopes = CLUSTER, description = "Paging cursor for Zendesk API is 
stored. Cursor is updated after each successful request.")
+@WritesAttributes({
+@WritesAttribute(attribute = RECORD_COUNT_ATTRIBUTE_NAME, description = 
"The number of records fetched by the processor.")})
+public class GetZendesk extends AbstractProcessor {
+
+static final int HTTP_TOO_MANY_REQUESTS = 429;
+static final String RECORD_COUNT_ATTRIBUTE_NAME = "record.count";
+

[GitHub] [nifi] bejancsaba commented on a diff in pull request #6411: NIFI-10460 GetZendesk processor

[GitBox]

bejancsaba commented on code in PR #6411:
URL: https://github.com/apache/nifi/pull/6411#discussion_r980149590


##
nifi-nar-bundles/nifi-zendesk-bundle/nifi-zendesk-processors/src/main/java/org/apache/nifi/processors/zendesk/GetZendesk.java:
##
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.zendesk;
+
+import static com.fasterxml.jackson.core.JsonEncoding.UTF8;
+import static com.fasterxml.jackson.core.JsonToken.FIELD_NAME;
+import static com.fasterxml.jackson.core.JsonToken.VALUE_NULL;
+import static java.lang.String.format;
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static java.util.Arrays.asList;
+import static java.util.Base64.getEncoder;
+import static java.util.Collections.singleton;
+import static java.util.Collections.singletonMap;
+import static java.util.Collections.unmodifiableList;
+import static 
org.apache.nifi.annotation.behavior.InputRequirement.Requirement.INPUT_FORBIDDEN;
+import static org.apache.nifi.components.state.Scope.CLUSTER;
+import static 
org.apache.nifi.expression.ExpressionLanguageScope.FLOWFILE_ATTRIBUTES;
+import static 
org.apache.nifi.processor.util.StandardValidators.NON_BLANK_VALIDATOR;
+import static 
org.apache.nifi.processor.util.StandardValidators.NON_EMPTY_VALIDATOR;
+import static 
org.apache.nifi.processor.util.StandardValidators.POSITIVE_LONG_VALIDATOR;
+import static 
org.apache.nifi.processors.zendesk.GetZendesk.RECORD_COUNT_ATTRIBUTE_NAME;
+import static org.apache.nifi.util.StringUtils.isNotBlank;
+import static org.apache.nifi.web.client.api.HttpResponseStatus.OK;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly

Review Comment:
   Thanks for the context



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (NIFI-10549) NiFi doesn't start if include-ranger profile is specified

[Matt Burgess (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609531#comment-17609531
 ] 

Matt Burgess commented on NIFI-10549:
-

Also interesting to note, if `-Ddir-only` is specified, it overrides the 
ranger.xml descriptor so you cannot have `-Ddir-only` specified along with 
`-Pinclude-ranger`

> NiFi doesn't start if include-ranger profile is specified
> -
>
> Key: NIFI-10549
> URL: https://issues.apache.org/jira/browse/NIFI-10549
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.18.0
>
>
> If the `-Pinclude-ranger` profile is included in the Maven build command for 
> NiFi, nifi-ranger-resources.jar and all its dependencies are put into lib, 
> causing the following error on startup:
> Failure to launch NiFi java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/JsonNode at 
> org.apache.nifi.util.hive.HiveJdbcCommon.(HiveJdbcCommon.java:92) at 
> org.apache.nifi.processors.hive.SelectHive3QL.(SelectHive3QL.java:272)
>  at java.lang.Class.forName0(Native Method) at 
> java.lang.Class.forName(Class.java:348) at 
> org.apache.nifi.nar.StandardExtensionDiscoveringManager.getClass(StandardExtensionDiscoveringManager.java:338)
>  at 
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:102)
>  at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:66) 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:795) at 
> org.apache.nifi.NiFi.(NiFi.java:172) at 
> org.apache.nifi.NiFi.(NiFi.java:83) at 
> org.apache.nifi.NiFi.main(NiFi.java:332) Caused by: 
> java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.JsonNode at 
> java.net.URLClassLoader.findClass(URLClassLoader.java:387) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:418) at 
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 11 common frames 
> omitted
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [nifi] ferencerdei commented on a diff in pull request #6438: NIFI-10531 Add supported operations to c2 heartbeat

[GitBox]

ferencerdei commented on code in PR #6438:
URL: https://github.com/apache/nifi/pull/6438#discussion_r980142584


##
c2/c2-client-bundle/c2-client-service/src/main/java/org/apache/nifi/c2/client/service/ManifestHashProvider.java:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.c2.client.service;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectOutputStream;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.c2.protocol.api.SupportedOperation;
+import org.apache.nifi.c2.protocol.component.api.Bundle;
+
+public class ManifestHashProvider {
+private volatile String currentBundles = null;

Review Comment:
   You are right, seems the threadpool size is 1 for the executor service so 
it's safe to remove this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (NIFI-10549) NiFi doesn't start if include-ranger profile is specified

[Matt Burgess (Jira)]

Matt Burgess created NIFI-10549:
---

 Summary: NiFi doesn't start if include-ranger profile is specified
 Key: NIFI-10549
 URL: https://issues.apache.org/jira/browse/NIFI-10549
 Project: Apache NiFi
  Issue Type: Bug
  Components: Tools and Build
Reporter: Matt Burgess
 Fix For: 1.18.0


If the `-Pinclude-ranger` profile is included in the Maven build command for 
NiFi, nifi-ranger-resources.jar and all its dependencies are put into lib, 
causing the following error on startup:

Failure to launch NiFi java.lang.NoClassDefFoundError: 
com/fasterxml/jackson/databind/JsonNode at 
org.apache.nifi.util.hive.HiveJdbcCommon.(HiveJdbcCommon.java:92) at 
org.apache.nifi.processors.hive.SelectHive3QL.(SelectHive3QL.java:272) 
at java.lang.Class.forName0(Native Method) at 
java.lang.Class.forName(Class.java:348) at 
org.apache.nifi.nar.StandardExtensionDiscoveringManager.getClass(StandardExtensionDiscoveringManager.java:338)
 at 
org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:102)
 at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:66) 
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:795) at 
org.apache.nifi.NiFi.(NiFi.java:172) at 
org.apache.nifi.NiFi.(NiFi.java:83) at 
org.apache.nifi.NiFi.main(NiFi.java:332) Caused by: 
java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.JsonNode at 
java.net.URLClassLoader.findClass(URLClassLoader.java:387) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:418) at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 11 common frames 
omitted

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10549) NiFi doesn't start if include-ranger profile is specified

[Matt Burgess (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609530#comment-17609530
 ] 

Matt Burgess commented on NIFI-10549:
-

This is a regression caused by NIFI-10406 where the asterisk need not be added 
as the group is specified.

> NiFi doesn't start if include-ranger profile is specified
> -
>
> Key: NIFI-10549
> URL: https://issues.apache.org/jira/browse/NIFI-10549
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Matt Burgess
>Priority: Major
> Fix For: 1.18.0
>
>
> If the `-Pinclude-ranger` profile is included in the Maven build command for 
> NiFi, nifi-ranger-resources.jar and all its dependencies are put into lib, 
> causing the following error on startup:
> Failure to launch NiFi java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/JsonNode at 
> org.apache.nifi.util.hive.HiveJdbcCommon.(HiveJdbcCommon.java:92) at 
> org.apache.nifi.processors.hive.SelectHive3QL.(SelectHive3QL.java:272)
>  at java.lang.Class.forName0(Native Method) at 
> java.lang.Class.forName(Class.java:348) at 
> org.apache.nifi.nar.StandardExtensionDiscoveringManager.getClass(StandardExtensionDiscoveringManager.java:338)
>  at 
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:102)
>  at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:66) 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:795) at 
> org.apache.nifi.NiFi.(NiFi.java:172) at 
> org.apache.nifi.NiFi.(NiFi.java:83) at 
> org.apache.nifi.NiFi.main(NiFi.java:332) Caused by: 
> java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.JsonNode at 
> java.net.URLClassLoader.findClass(URLClassLoader.java:387) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:418) at 
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 11 common frames 
> omitted
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (NIFI-10549) NiFi doesn't start if include-ranger profile is specified

[Matt Burgess (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess reassigned NIFI-10549:
---

Assignee: Matt Burgess

> NiFi doesn't start if include-ranger profile is specified
> -
>
> Key: NIFI-10549
> URL: https://issues.apache.org/jira/browse/NIFI-10549
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.18.0
>
>
> If the `-Pinclude-ranger` profile is included in the Maven build command for 
> NiFi, nifi-ranger-resources.jar and all its dependencies are put into lib, 
> causing the following error on startup:
> Failure to launch NiFi java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/JsonNode at 
> org.apache.nifi.util.hive.HiveJdbcCommon.(HiveJdbcCommon.java:92) at 
> org.apache.nifi.processors.hive.SelectHive3QL.(SelectHive3QL.java:272)
>  at java.lang.Class.forName0(Native Method) at 
> java.lang.Class.forName(Class.java:348) at 
> org.apache.nifi.nar.StandardExtensionDiscoveringManager.getClass(StandardExtensionDiscoveringManager.java:338)
>  at 
> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:102)
>  at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:66) 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:795) at 
> org.apache.nifi.NiFi.(NiFi.java:172) at 
> org.apache.nifi.NiFi.(NiFi.java:83) at 
> org.apache.nifi.NiFi.main(NiFi.java:332) Caused by: 
> java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.JsonNode at 
> java.net.URLClassLoader.findClass(URLClassLoader.java:387) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:418) at 
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 11 common frames 
> omitted
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10547) NiFi cluster is unstable with error "Administrative Yield Duration"

[Joe Witt (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609528#comment-17609528
 ] 

Joe Witt commented on NIFI-10547:
-

Hello.  We will need more details about the configuration but specifically we 
will need the stack traces.  This looks like a grep for ERROR in the logs but 
that will only take the first thing.  We need the stacks that follow.  There is 
clearly an issue with the Cassandra QL processor.  This being NiFi 1.8.0 it is 
likely the solution will be to upgrade.

Thanks

> NiFi cluster is unstable with error "Administrative Yield Duration"
> ---
>
> Key: NIFI-10547
> URL: https://issues.apache.org/jira/browse/NIFI-10547
> Project: Apache NiFi
>  Issue Type: Task
>Affects Versions: 1.8.0
> Environment: Prod
>Reporter: Anoop Ajit
>Priority: Major
>
> Nifi cluster is refusing to be stable and is throwing various errors like 
> "Too many Open files" and "Administrative Yield Duration". 
>  
> Below is the error logs:
>  
> 2022-09-26 21:25:25,745 ERROR [Timer-Driven Process Thread-83] 
> o.a.n.p.cassandra.QueryCassandra 
> QueryCassandra[id=daa02218-016b-1000--f4865b2f] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.reflect.InvocationTargetException: 
> java.lang.reflect.InvocationTargetException
> 2022-09-26 21:25:25,775 ERROR [Timer-Driven Process Thread-139] 
> o.a.n.p.cassandra.PutCassandraQL 
> PutCassandraQL[id=d6ad5e1c-eec8-3e77-9636-247017134eee] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.reflect.InvocationTargetException: 
> java.lang.reflect.InvocationTargetException
> 2022-09-26 21:25:57,865 ERROR [Timer-Driven Process Thread-31] 
> o.a.n.p.cassandra.QueryCassandra 
> QueryCassandra[id=daa02218-016b-1000--f4865b2f] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.reflect.InvocationTargetException: 
> java.lang.reflect.InvocationTargetException
> 2022-09-26 21:25:57,974 ERROR [Timer-Driven Process Thread-44] 
> o.a.n.p.cassandra.PutCassandraQL 
> PutCassandraQL[id=f02c3fb7-3a9c-147c-9bc1-d30b6ba15183] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.reflect.InvocationTargetException: 
> java.lang.reflect.InvocationTargetException
> 2022-09-26 21:25:58,006 ERROR [Timer-Driven Process Thread-164] 
> o.a.n.p.cassandra.PutCassandraQL 
> PutCassandraQL[id=d6ad5e1c-eec8-3e77-9636-247017134eee] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.reflect.InvocationTargetException: 
> java.lang.reflect.InvocationTargetException
> 2022-09-26 21:26:20,039 ERROR [Timer-Driven Process Thread-72] 
> o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
> StandardProcessGroup[identifier=318ec9e8-016b-1000--c852c4db] with 
> Flow Registry because could not retrieve version 1 of flow with identifier 
> dd7b8feb-4f95-495e-980f-05d89641daff in bucket 
> 49b79584-8e25-4e3b-80cc-bf2b6ff4a452
> 2022-09-26 21:26:20,040 ERROR [Timer-Driven Process Thread-72] 
> o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
> StandardProcessGroup[identifier=3190cbfe-016b-1000--ab2b0f70] with 
> Flow Registry because could not retrieve version 1 of flow with identifier 
> 70323c57-4dc0-4157-ba01-7efbef1a1e1b in bucket 
> 0a33dfc5-dd66-49d7-977b-e82a56bcc4df
> 2022-09-26 21:26:20,041 ERROR [Timer-Driven Process Thread-72] 
> o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
> StandardProcessGroup[identifier=f59f32d1-d71f-1cbc-86ad-9242e95745ad] with 
> Flow Registry because could not retrieve version 1 of flow with identifier 
> 4dc233a0-d21e-4a41-b271-80d3f4d9310f in bucket 
> 0a33dfc5-dd66-49d7-977b-e82a56bcc4df
> 2022-09-26 21:26:20,042 ERROR [Timer-Driven Process Thread-72] 
> o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
> StandardProcessGroup[identifier=b4ea3324-079f-1226-b6b6-7e93c8b1318d] with 
> Flow Registry because could not retrieve version 1 of flow with identifier 
> 67db4103-271f-43c8-8bc6-68c7729f3504 in bucket 

[GitHub] [nifi] ferencerdei commented on a diff in pull request #6438: NIFI-10531 Add supported operations to c2 heartbeat

[GitBox]

ferencerdei commented on code in PR #6438:
URL: https://github.com/apache/nifi/pull/6438#discussion_r980127950


##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/c2/C2NifiClientService.java:
##
@@ -90,15 +96,18 @@ public C2NifiClientService(final NiFiProperties 
niFiProperties, final FlowContro
 this.heartbeatPeriod = clientConfig.getHeartbeatPeriod();
 this.flowController = flowController;
 C2HttpClient client = new C2HttpClient(clientConfig, new 
C2JacksonSerializer());
-C2HeartbeatFactory heartbeatFactory = new 
C2HeartbeatFactory(clientConfig, flowIdHolder);
+C2HeartbeatFactory heartbeatFactory = new 
C2HeartbeatFactory(clientConfig, flowIdHolder, new ManifestHashProvider());
+OperandPropertiesProvider emptyOperandPropertiesProvider = new 
EmptyOperandPropertiesProvider();

Review Comment:
   Yes, for now, it doesn't have too much added value because neither of the 
supported operations has any additional properties, but in the near future, I'm 
planning to add the update properties command where I think it'll be better to 
pass these to the C2 client.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] ferencerdei commented on a diff in pull request #6438: NIFI-10531 Add supported operations to c2 heartbeat

[GitBox]

ferencerdei commented on code in PR #6438:
URL: https://github.com/apache/nifi/pull/6438#discussion_r980112822


##
c2/c2-client-bundle/c2-client-service/src/main/java/org/apache/nifi/c2/client/service/C2HeartbeatFactory.java:
##
@@ -106,6 +106,17 @@ private AgentInfo getAgentInfo(AgentRepositories repos, 
RuntimeManifest manifest
 return agentInfo;
 }
 
+private Set getSupportedOperations(RuntimeManifest 
manifest) {
+Set supportedOperations;
+// supported operations has value only in case of minifi, therefore we 
return empty collection if

Review Comment:
   The RuntimeManifest is used by nifi-web module as well that's why I created 
a wrapper class. In case of NiFi this doesn't have any meaning for now, that's 
why I made this condition here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] briansolo1985 commented on a diff in pull request #6438: NIFI-10531 Add supported operations to c2 heartbeat

[GitBox]

briansolo1985 commented on code in PR #6438:
URL: https://github.com/apache/nifi/pull/6438#discussion_r979887567


##
c2/c2-client-bundle/c2-client-service/src/main/java/org/apache/nifi/c2/client/service/C2HeartbeatFactory.java:
##
@@ -106,6 +106,17 @@ private AgentInfo getAgentInfo(AgentRepositories repos, 
RuntimeManifest manifest
 return agentInfo;
 }
 
+private Set getSupportedOperations(RuntimeManifest 
manifest) {
+Set supportedOperations;
+// supported operations has value only in case of minifi, therefore we 
return empty collection if

Review Comment:
   Not sure if we should put minifi specific code here. I think if we make 
supportedOperations part of the C2 protocol, we should make it generic. In 
other words. we could extend the RuntimeManifest with the supportedOperations 
field and omit AgentManifest class. What do you think?



##
c2/c2-client-bundle/c2-client-service/src/main/java/org/apache/nifi/c2/client/service/ManifestHashProvider.java:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.c2.client.service;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectOutputStream;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.c2.protocol.api.SupportedOperation;
+import org.apache.nifi.c2.protocol.component.api.Bundle;
+
+public class ManifestHashProvider {
+private volatile String currentBundles = null;
+private volatile Set currentSupportedOperations = 
Collections.emptySet();
+private volatile Pair currentHashCodeManifestHashMapping 
= Pair.of(-1, null);

Review Comment:
   I know Pair represents a mapping relationship between hash and manifest 
hash, but for me it makes the code less comprehensible. We could define the 
content of the pair as separate class members.



##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/c2/C2NifiClientService.java:
##
@@ -90,15 +96,18 @@ public C2NifiClientService(final NiFiProperties 
niFiProperties, final FlowContro
 this.heartbeatPeriod = clientConfig.getHeartbeatPeriod();
 this.flowController = flowController;
 C2HttpClient client = new C2HttpClient(clientConfig, new 
C2JacksonSerializer());
-C2HeartbeatFactory heartbeatFactory = new 
C2HeartbeatFactory(clientConfig, flowIdHolder);
+C2HeartbeatFactory heartbeatFactory = new 
C2HeartbeatFactory(clientConfig, flowIdHolder, new ManifestHashProvider());
+OperandPropertiesProvider emptyOperandPropertiesProvider = new 
EmptyOperandPropertiesProvider();

Review Comment:
   A question: do we need to inject operandPropertiesProvider from here to keep 
the C2 layer decoupled from Minifi?
   My feeling is that this class start to get a bit crowded, and as a glue 
between Minifi and C2 I expect it to have even more logic in the future. I 
think we should think about how we could extract logic into separate classes 
(not strictly in the PR)



##
c2/c2-client-bundle/c2-client-service/src/main/java/org/apache/nifi/c2/client/service/ManifestHashProvider.java:
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the Lice

[jira] [Comment Edited] (NIFI-10198) PutDatabaseRecord only checks required colums for 'Fail on Unmatched Columns'

[He Jiang (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609518#comment-17609518
 ] 

He Jiang edited comment on NIFI-10198 at 9/26/22 2:13 PM:
--

@sbylemans Another tricky case that I often used is adding a datetime column in 
mysql table with the default value of CURRENT_TIMESTAMP  to track the creation 
time of a record.

Generally speaking, any column with default value of a table could be skipped 
safely for putting record in.

So I just recommend to revise the description to "If an incoming record does 
not have a field mapping for *all required columns* of the database table, this 
property specifies how to handle the situation." 

Any more thoughts?


was (Author: hejiang):
@sbylemans Another tricky case that I often used is adding a datetime column in 
mysql table with the default value of CURRENT_TIMESTAMP  to track the creation 
time of a record.

Generally speaking, any column with default value of a table could be skipped 
safely for putting record in.

So I just recommend to revise the description to "If an incoming record does 
not have a field mapping for *all required columns* of the database table, this 
property specifies how to handle the situation." 

Any more thoughtw?

> PutDatabaseRecord only checks required colums for 'Fail on Unmatched Columns'
> -
>
> Key: NIFI-10198
> URL: https://issues.apache.org/jira/browse/NIFI-10198
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.15.3
>Reporter: Sander Bylemans
>Priority: Critical
>  Labels: patch
>
> The PutDatabaseRecord processor has a method called 
> {{checkValuesForRequiredColumns}}:
> {code:java}
> private void checkValuesForRequiredColumns(RecordSchema recordSchema, 
> TableSchema tableSchema, DMLSettings settings) {
> final Set normalizedFieldNames = 
> getNormalizedColumnNames(recordSchema, settings.translateFieldNames);
> for (final String requiredColName : 
> tableSchema.getRequiredColumnNames()) {
> final String normalizedColName = 
> normalizeColumnName(requiredColName, settings.translateFieldNames);
> if (!normalizedFieldNames.contains(normalizedColName)) {
> String missingColMessage = "Record does not have a value for 
> the Required column '" + requiredColName + "'";
> if (settings.failUnmappedColumns) {
> getLogger().error(missingColMessage);
> throw new IllegalArgumentException(missingColMessage);
> } else if (settings.warningUnmappedColumns) {
> getLogger().warn(missingColMessage);
> }
> }
> }
> }
> {code}
> Which checks columns that are defined with a {{NOT NULL}} constraint. For the 
> {{Unmatched Column Behavior}} with a value {{Fail on Unmatched Columns}}, 
> this is not expected behavior as the description reads: "If an incoming 
> record does not have a field mapping for *all* of the database table's 
> columns, this property specifies how to handle the situation".
> Therefore the method should be adjusted so it checks all columns, not just 
> the "required" ones.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10198) PutDatabaseRecord only checks required colums for 'Fail on Unmatched Columns'

[He Jiang (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609518#comment-17609518
 ] 

He Jiang commented on NIFI-10198:
-

@sbylemans Another tricky case that I often used is adding a datetime column in 
mysql table with the default value of CURRENT_TIMESTAMP  to track the creation 
time of a record.

Generally speaking, any column with default value of a table could be skipped 
safely for putting record in.

So I just recommend to revise the description to "If an incoming record does 
not have a field mapping for *all required columns* of the database table, this 
property specifies how to handle the situation." 

Any more thoughtw?

> PutDatabaseRecord only checks required colums for 'Fail on Unmatched Columns'
> -
>
> Key: NIFI-10198
> URL: https://issues.apache.org/jira/browse/NIFI-10198
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.15.3
>Reporter: Sander Bylemans
>Priority: Critical
>  Labels: patch
>
> The PutDatabaseRecord processor has a method called 
> {{checkValuesForRequiredColumns}}:
> {code:java}
> private void checkValuesForRequiredColumns(RecordSchema recordSchema, 
> TableSchema tableSchema, DMLSettings settings) {
> final Set normalizedFieldNames = 
> getNormalizedColumnNames(recordSchema, settings.translateFieldNames);
> for (final String requiredColName : 
> tableSchema.getRequiredColumnNames()) {
> final String normalizedColName = 
> normalizeColumnName(requiredColName, settings.translateFieldNames);
> if (!normalizedFieldNames.contains(normalizedColName)) {
> String missingColMessage = "Record does not have a value for 
> the Required column '" + requiredColName + "'";
> if (settings.failUnmappedColumns) {
> getLogger().error(missingColMessage);
> throw new IllegalArgumentException(missingColMessage);
> } else if (settings.warningUnmappedColumns) {
> getLogger().warn(missingColMessage);
> }
> }
> }
> }
> {code}
> Which checks columns that are defined with a {{NOT NULL}} constraint. For the 
> {{Unmatched Column Behavior}} with a value {{Fail on Unmatched Columns}}, 
> this is not expected behavior as the description reads: "If an incoming 
> record does not have a field mapping for *all* of the database table's 
> columns, this property specifies how to handle the situation".
> Therefore the method should be adjusted so it checks all columns, not just 
> the "required" ones.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (NIFI-10548) Update Jetty From 9.4.48.v20220622 to 9.4.49.v20220914

[Mike R (Jira)]

Mike R created NIFI-10548:
-

 Summary: Update Jetty From 9.4.48.v20220622 to 9.4.49.v20220914
 Key: NIFI-10548
 URL: https://issues.apache.org/jira/browse/NIFI-10548
 Project: Apache NiFi
  Issue Type: Improvement
Affects Versions: 1.17.0
Reporter: Mike R


Update Jetty From 9.4.48.v20220622 to 9.4.49.v20220914



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (NIFI-10198) PutDatabaseRecord only checks required colums for 'Fail on Unmatched Columns'

[Sander Bylemans (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609509#comment-17609509
 ] 

Sander Bylemans commented on NIFI-10198:


[~hejiang] that's the easy way out. Are there any more that you can think of 
that don't need mapping? Because you could say a primary key is optional.

So that means adjusting the description to: "If an incoming record does not 
have a field mapping for *all* of the database table's columns, this property 
specifies how to handle the situation. The primary key is optional for the case 
of auto increment id's."

And then also changing the code to handle this.

> PutDatabaseRecord only checks required colums for 'Fail on Unmatched Columns'
> -
>
> Key: NIFI-10198
> URL: https://issues.apache.org/jira/browse/NIFI-10198
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.15.3
>Reporter: Sander Bylemans
>Priority: Critical
>  Labels: patch
>
> The PutDatabaseRecord processor has a method called 
> {{checkValuesForRequiredColumns}}:
> {code:java}
> private void checkValuesForRequiredColumns(RecordSchema recordSchema, 
> TableSchema tableSchema, DMLSettings settings) {
> final Set normalizedFieldNames = 
> getNormalizedColumnNames(recordSchema, settings.translateFieldNames);
> for (final String requiredColName : 
> tableSchema.getRequiredColumnNames()) {
> final String normalizedColName = 
> normalizeColumnName(requiredColName, settings.translateFieldNames);
> if (!normalizedFieldNames.contains(normalizedColName)) {
> String missingColMessage = "Record does not have a value for 
> the Required column '" + requiredColName + "'";
> if (settings.failUnmappedColumns) {
> getLogger().error(missingColMessage);
> throw new IllegalArgumentException(missingColMessage);
> } else if (settings.warningUnmappedColumns) {
> getLogger().warn(missingColMessage);
> }
> }
> }
> }
> {code}
> Which checks columns that are defined with a {{NOT NULL}} constraint. For the 
> {{Unmatched Column Behavior}} with a value {{Fail on Unmatched Columns}}, 
> this is not expected behavior as the description reads: "If an incoming 
> record does not have a field mapping for *all* of the database table's 
> columns, this property specifies how to handle the situation".
> Therefore the method should be adjusted so it checks all columns, not just 
> the "required" ones.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (NIFI-10547) NiFi cluster is unstable with error "Administrative Yield Duration"

[Anoop Ajit (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Ajit updated NIFI-10547:
--
Issue Type: Task  (was: Bug)

> NiFi cluster is unstable with error "Administrative Yield Duration"
> ---
>
> Key: NIFI-10547
> URL: https://issues.apache.org/jira/browse/NIFI-10547
> Project: Apache NiFi
>  Issue Type: Task
>Affects Versions: 1.8.0
> Environment: Prod
>Reporter: Anoop Ajit
>Priority: Major
>
> Nifi cluster is refusing to be stable and is throwing various errors like 
> "Too many Open files" and "Administrative Yield Duration". 
>  
> Below is the error logs:
>  
> 2022-09-26 21:25:25,745 ERROR [Timer-Driven Process Thread-83] 
> o.a.n.p.cassandra.QueryCassandra 
> QueryCassandra[id=daa02218-016b-1000--f4865b2f] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.reflect.InvocationTargetException: 
> java.lang.reflect.InvocationTargetException
> 2022-09-26 21:25:25,775 ERROR [Timer-Driven Process Thread-139] 
> o.a.n.p.cassandra.PutCassandraQL 
> PutCassandraQL[id=d6ad5e1c-eec8-3e77-9636-247017134eee] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.reflect.InvocationTargetException: 
> java.lang.reflect.InvocationTargetException
> 2022-09-26 21:25:57,865 ERROR [Timer-Driven Process Thread-31] 
> o.a.n.p.cassandra.QueryCassandra 
> QueryCassandra[id=daa02218-016b-1000--f4865b2f] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.reflect.InvocationTargetException: 
> java.lang.reflect.InvocationTargetException
> 2022-09-26 21:25:57,974 ERROR [Timer-Driven Process Thread-44] 
> o.a.n.p.cassandra.PutCassandraQL 
> PutCassandraQL[id=f02c3fb7-3a9c-147c-9bc1-d30b6ba15183] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.reflect.InvocationTargetException: 
> java.lang.reflect.InvocationTargetException
> 2022-09-26 21:25:58,006 ERROR [Timer-Driven Process Thread-164] 
> o.a.n.p.cassandra.PutCassandraQL 
> PutCassandraQL[id=d6ad5e1c-eec8-3e77-9636-247017134eee] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elapsed. Failure is due to 
> java.lang.reflect.InvocationTargetException: 
> java.lang.reflect.InvocationTargetException
> 2022-09-26 21:26:20,039 ERROR [Timer-Driven Process Thread-72] 
> o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
> StandardProcessGroup[identifier=318ec9e8-016b-1000--c852c4db] with 
> Flow Registry because could not retrieve version 1 of flow with identifier 
> dd7b8feb-4f95-495e-980f-05d89641daff in bucket 
> 49b79584-8e25-4e3b-80cc-bf2b6ff4a452
> 2022-09-26 21:26:20,040 ERROR [Timer-Driven Process Thread-72] 
> o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
> StandardProcessGroup[identifier=3190cbfe-016b-1000--ab2b0f70] with 
> Flow Registry because could not retrieve version 1 of flow with identifier 
> 70323c57-4dc0-4157-ba01-7efbef1a1e1b in bucket 
> 0a33dfc5-dd66-49d7-977b-e82a56bcc4df
> 2022-09-26 21:26:20,041 ERROR [Timer-Driven Process Thread-72] 
> o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
> StandardProcessGroup[identifier=f59f32d1-d71f-1cbc-86ad-9242e95745ad] with 
> Flow Registry because could not retrieve version 1 of flow with identifier 
> 4dc233a0-d21e-4a41-b271-80d3f4d9310f in bucket 
> 0a33dfc5-dd66-49d7-977b-e82a56bcc4df
> 2022-09-26 21:26:20,042 ERROR [Timer-Driven Process Thread-72] 
> o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
> StandardProcessGroup[identifier=b4ea3324-079f-1226-b6b6-7e93c8b1318d] with 
> Flow Registry because could not retrieve version 1 of flow with identifier 
> 67db4103-271f-43c8-8bc6-68c7729f3504 in bucket 
> 0a33dfc5-dd66-49d7-977b-e82a56bcc4df
> 2022-09-26 21:26:28,174 ERROR [Timer-Driven Process Thread-220] 
> o.a.n.p.cassandra.QueryCassandra 
> QueryCassandra[id=daa02218-016b-1000--f4865b2f] Failed to properly 
> initialize Processor. If still scheduled to run, NiFi will attempt to 
> initialize and run the Processor again after the 'Administrative Yield 
> Duration' has elap

[jira] [Created] (NIFI-10547) NiFi cluster is unstable with error "Administrative Yield Duration"

[Anoop Ajit (Jira)]

Anoop Ajit created NIFI-10547:
-

 Summary: NiFi cluster is unstable with error "Administrative Yield 
Duration"
 Key: NIFI-10547
 URL: https://issues.apache.org/jira/browse/NIFI-10547
 Project: Apache NiFi
  Issue Type: Bug
Affects Versions: 1.8.0
 Environment: Prod
Reporter: Anoop Ajit


Nifi cluster is refusing to be stable and is throwing various errors like "Too 
many Open files" and "Administrative Yield Duration". 

 

Below is the error logs:

 

2022-09-26 21:25:25,745 ERROR [Timer-Driven Process Thread-83] 
o.a.n.p.cassandra.QueryCassandra 
QueryCassandra[id=daa02218-016b-1000--f4865b2f] Failed to properly 
initialize Processor. If still scheduled to run, NiFi will attempt to 
initialize and run the Processor again after the 'Administrative Yield 
Duration' has elapsed. Failure is due to 
java.lang.reflect.InvocationTargetException: 
java.lang.reflect.InvocationTargetException
2022-09-26 21:25:25,775 ERROR [Timer-Driven Process Thread-139] 
o.a.n.p.cassandra.PutCassandraQL 
PutCassandraQL[id=d6ad5e1c-eec8-3e77-9636-247017134eee] Failed to properly 
initialize Processor. If still scheduled to run, NiFi will attempt to 
initialize and run the Processor again after the 'Administrative Yield 
Duration' has elapsed. Failure is due to 
java.lang.reflect.InvocationTargetException: 
java.lang.reflect.InvocationTargetException
2022-09-26 21:25:57,865 ERROR [Timer-Driven Process Thread-31] 
o.a.n.p.cassandra.QueryCassandra 
QueryCassandra[id=daa02218-016b-1000--f4865b2f] Failed to properly 
initialize Processor. If still scheduled to run, NiFi will attempt to 
initialize and run the Processor again after the 'Administrative Yield 
Duration' has elapsed. Failure is due to 
java.lang.reflect.InvocationTargetException: 
java.lang.reflect.InvocationTargetException
2022-09-26 21:25:57,974 ERROR [Timer-Driven Process Thread-44] 
o.a.n.p.cassandra.PutCassandraQL 
PutCassandraQL[id=f02c3fb7-3a9c-147c-9bc1-d30b6ba15183] Failed to properly 
initialize Processor. If still scheduled to run, NiFi will attempt to 
initialize and run the Processor again after the 'Administrative Yield 
Duration' has elapsed. Failure is due to 
java.lang.reflect.InvocationTargetException: 
java.lang.reflect.InvocationTargetException
2022-09-26 21:25:58,006 ERROR [Timer-Driven Process Thread-164] 
o.a.n.p.cassandra.PutCassandraQL 
PutCassandraQL[id=d6ad5e1c-eec8-3e77-9636-247017134eee] Failed to properly 
initialize Processor. If still scheduled to run, NiFi will attempt to 
initialize and run the Processor again after the 'Administrative Yield 
Duration' has elapsed. Failure is due to 
java.lang.reflect.InvocationTargetException: 
java.lang.reflect.InvocationTargetException
2022-09-26 21:26:20,039 ERROR [Timer-Driven Process Thread-72] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=318ec9e8-016b-1000--c852c4db] with Flow 
Registry because could not retrieve version 1 of flow with identifier 
dd7b8feb-4f95-495e-980f-05d89641daff in bucket 
49b79584-8e25-4e3b-80cc-bf2b6ff4a452
2022-09-26 21:26:20,040 ERROR [Timer-Driven Process Thread-72] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=3190cbfe-016b-1000--ab2b0f70] with Flow 
Registry because could not retrieve version 1 of flow with identifier 
70323c57-4dc0-4157-ba01-7efbef1a1e1b in bucket 
0a33dfc5-dd66-49d7-977b-e82a56bcc4df
2022-09-26 21:26:20,041 ERROR [Timer-Driven Process Thread-72] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=f59f32d1-d71f-1cbc-86ad-9242e95745ad] with Flow 
Registry because could not retrieve version 1 of flow with identifier 
4dc233a0-d21e-4a41-b271-80d3f4d9310f in bucket 
0a33dfc5-dd66-49d7-977b-e82a56bcc4df
2022-09-26 21:26:20,042 ERROR [Timer-Driven Process Thread-72] 
o.a.nifi.groups.StandardProcessGroup Failed to synchronize 
StandardProcessGroup[identifier=b4ea3324-079f-1226-b6b6-7e93c8b1318d] with Flow 
Registry because could not retrieve version 1 of flow with identifier 
67db4103-271f-43c8-8bc6-68c7729f3504 in bucket 
0a33dfc5-dd66-49d7-977b-e82a56bcc4df
2022-09-26 21:26:28,174 ERROR [Timer-Driven Process Thread-220] 
o.a.n.p.cassandra.QueryCassandra 
QueryCassandra[id=daa02218-016b-1000--f4865b2f] Failed to properly 
initialize Processor. If still scheduled to run, NiFi will attempt to 
initialize and run the Processor again after the 'Administrative Yield 
Duration' has elapsed. Failure is due to 
java.lang.reflect.InvocationTargetException: 
java.lang.reflect.InvocationTargetException
2022-09-26 21:26:28,259 ERROR [Timer-Driven Process Thread-145] 
o.a.n.p.cassandra.PutCassandraQL 
PutCassandraQL[id=f02c3fb7-3a9c-147c-9bc1-d30b6ba15183] Failed to properly 
initialize Processor. If still scheduled to run, NiFi will attempt to 
initialize and run the Processor 

[GitHub] [nifi] mcgilman commented on a diff in pull request #6437: NIFI-10538: UI updates for making RegistryClient an extension point

[GitBox]

mcgilman commented on code in PR #6437:
URL: https://github.com/apache/nifi/pull/6437#discussion_r980008876


##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-controller-service.js:
##
@@ -670,6 +702,44 @@
 var providerItem = 
$('').append(providerState).append(providerBulletins).append(parameterProviderLink).append(providerType);
 
 providers.append(providerItem);
+} else if (referencingComponent.referenceType === 
'FlowRegistryClient') {
+var registryLink = $('').text(referencingComponent.name).on('click', function () {
+var registryGrid = 
$('#registries-table').data('gridInstance');
+var registryData = registryGrid.getData();
+
+// select the selected row
+var row = 
registryData.getRowById(referencingComponent.id);
+registryGrid.setSelectedRows([row]);
+registryGrid.scrollRowIntoView(row);
+
+// select the reporting task tab
+$('#settings-tabs').find('li:nth-child(4)').click();
+
+// close the dialog and shell
+referenceContainer.closest('.dialog').modal('hide');
+});
+
+// registry state - used to show the validation errors
+var registryState = $('').addClass(referencingComponent.id + 'state');
+
+if 
(nfCommon.isEmpty(referencingComponent.validationErrors)) {
+registryState.hide();
+} else {
+updateValidationErrors(registryState, 
referencingComponent);
+}
+
+// type
+var registryType = $('').text(nfCommon.substringAfterLast(referencingComponent.type,
 '.'));
+
+// active thread count
+var registryActiveThreadCount = $('').addClass(referencingComponent.id
 + '-active-threads');
+if 
(nfCommon.isDefinedAndNotNull(referencingComponent.activeThreadCount) && 
referencingComponent.activeThreadCount > 0) {
+registryActiveThreadCount.text('(' + 
referencingComponent.activeThreadCount + ')');
+}
+
+// registry item
+var registryItem = 
$('').append(registryState).append(registryLink).append(registryType).append(registryActiveThreadCount);
+tasks.append(registryItem);

Review Comment:
   These registry clients are being added to the Reporting Tasks section. 
Please create a dedicated section to Flow Registry Clients.



##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-controller-service.js:
##
@@ -670,6 +702,44 @@
 var providerItem = 
$('').append(providerState).append(providerBulletins).append(parameterProviderLink).append(providerType);
 
 providers.append(providerItem);
+} else if (referencingComponent.referenceType === 
'FlowRegistryClient') {
+var registryLink = $('').text(referencingComponent.name).on('click', function () {
+var registryGrid = 
$('#registries-table').data('gridInstance');
+var registryData = registryGrid.getData();
+
+// select the selected row
+var row = 
registryData.getRowById(referencingComponent.id);
+registryGrid.setSelectedRows([row]);
+registryGrid.scrollRowIntoView(row);
+
+// select the reporting task tab
+$('#settings-tabs').find('li:nth-child(4)').click();
+
+// close the dialog and shell
+referenceContainer.closest('.dialog').modal('hide');
+});
+
+// registry state - used to show the validation errors
+var registryState = $('').addClass(referencingComponent.id + 'state');
+
+if 
(nfCommon.isEmpty(referencingComponent.validationErrors)) {
+registryState.hide();
+} else {
+updateValidationErrors(registryState, 
referencingComponent);
+}
+
+// type
+var registryType = $('').text(nfCommon.substringAfterLast(referencingComponent.type,
 '.'));
+
+// active thread count
+var registryActiveThreadCount = $('').addClass(referencingComponent.id
 + '-active-threads');
+if 
(nfCommon.isDefinedAndNotNull(referencingComponent.activeThreadCount) && 
referencingComponent.activeThreadCount

[GitHub] [nifi] mcgilman commented on a diff in pull request #6437: NIFI-10538: UI updates for making RegistryClient an extension point

[GitBox]

mcgilman commented on code in PR #6437:
URL: https://github.com/apache/nifi/pull/6437#discussion_r979979750


##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-controller-service.js:
##
@@ -257,10 +257,12 @@
 // reload
 nfParameterProvider.reload(reference.id);
 
+} else if (reference.referenceType === 'FlowRegistryClient') {

Review Comment:
   I think something may have been mixed up during a rebase here.



##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-controller-service.js:
##
@@ -402,6 +404,36 @@
 });
 };
 
+var updateValidationErrors = function (validationErrorIcon, 
referencingComponent) {

Review Comment:
   This function appears to be a duplicate. Likely another mixed up with the 
rebase. Also, there's another condition that may have been missed. Search for 
'ParameterProvider' to find other instances.



##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-flow-version.js:
##
@@ -206,7 +206,8 @@
 registries.push({
 text: registry.name,
 value: registry.id,
-description: nfCommon.escapeHtml(registry.description)
+description: nfCommon.escapeHtml(registry.description),
+disabled: registry.validationStatus !== 'VALID' // 
TODO: verify this logic is correct

Review Comment:
   This looks correct but we should check with @simonbence or @markap14 who are 
a little more familiar with the backend effort [1].
   
   [1] #6433 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

adamdebreceni commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r979899914


##
libminifi/include/core/flow/Node.h:
##
@@ -0,0 +1,143 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include "nonstd/expected.hpp"
+
+namespace org::apache::nifi::minifi::core::flow {
+
+class Node {
+ public:
+  struct Cursor {
+int line;
+int column;
+int pos;
+  };
+
+  class Impl;
+  class Iterator {
+   public:
+class Value;
+
+class Impl {
+ public:
+  virtual Impl& operator++() = 0;
+  virtual bool operator==(const Impl& other) const = 0;
+  virtual Value operator*() const = 0;
+  bool operator!=(const Impl& other) const {return !(*this == other);}
+
+  virtual std::unique_ptr clone() const = 0;
+  virtual ~Impl() = default;
+};
+
+Iterator& operator++() {
+  impl_->operator++();
+  return *this;
+}
+
+explicit Iterator(std::unique_ptr impl) : impl_(std::move(impl)) {}
+Iterator(const Iterator& other): impl_(other.impl_->clone()) {}
+Iterator(Iterator&&) = default;
+Iterator& operator=(const Iterator& other) {
+  if (this == &other) {
+return *this;
+  }
+  impl_ = other.impl_->clone();
+  return *this;
+}
+Iterator& operator=(Iterator&&) = default;
+
+bool operator==(const Iterator& other) const {return 
impl_->operator==(*other.impl_);}
+bool operator!=(const Iterator& other) const {return !(*this == other);}
+
+Value operator*() const;
+
+   private:
+std::unique_ptr impl_;
+  };
+
+  class Impl {
+   public:
+virtual explicit operator bool() const = 0;
+virtual bool isSequence() const = 0;
+virtual bool isMap() const = 0;
+virtual bool isNull() const = 0;
+virtual bool isScalar() const = 0;
+
+virtual nonstd::expected getString() 
const = 0;
+virtual nonstd::expected getInt() const = 0;
+virtual nonstd::expected getUInt() const 
= 0;
+virtual nonstd::expected getBool() const = 0;
+virtual nonstd::expected getInt64() const = 0;
+virtual nonstd::expected getUInt64() const = 
0;
+
+virtual std::string getDebugString() const = 0;
+
+virtual size_t size() const = 0;
+virtual Iterator begin() const = 0;
+virtual Iterator end() const = 0;
+virtual Node operator[](std::string_view key) const = 0;
+
+virtual std::optional getCursor() const {return std::nullopt;}
+
+virtual ~Impl() = default;
+  };
+
+  Node() = default;
+  explicit Node(std::shared_ptr impl): impl_(std::move(impl)) {}
+
+  explicit operator bool() const {return impl_->operator bool();}
+  bool isSequence() const {return impl_->isSequence();}
+  bool isMap() const {return impl_->isMap();}
+  bool isNull() const {return impl_->isNull();}
+  bool isScalar() const {return impl_->isScalar();}
+
+  nonstd::expected getString() const {return 
impl_->getString();}
+  nonstd::expected getBool() const {return 
impl_->getBool();}
+  nonstd::expected getInt() const {return 
impl_->getInt();}
+  nonstd::expected getUInt() const {return 
impl_->getUInt();}
+  nonstd::expected getInt64() const {return 
impl_->getInt64();}
+  nonstd::expected getUInt64() const {return 
impl_->getUInt64();}
+
+  // return a string representation of the node (need not to be deserializable)
+  std::string getDebugString() const {return impl_->getDebugString();}
+
+  size_t size() const {return impl_->size();}
+  size_t empty() const {
+return size() == 0;
+  }
+  Iterator begin() const {return impl_->begin();}
+  Iterator end() const {return impl_->end();}
+  Node operator[](std::string_view key) const {return impl_->operator[](key);}
+
+  std::optional getCursor() const {return impl_->getCursor();}
+
+ private:
+  std::shared_ptr impl_;
+};
+
+class Node::Iterator::Value : public Node, public std::pair {
+ public:
+  Value(Node node, Node key, Node value): Node{std::move(node)}, 
std::pair{std::move(key), std::move(value)} {}
+};

Review Comment:
   I wanted to mimic the `yaml-cpp`'s implementation (`node/iterator.h`) 
allowing uniform iteration on both maps and sequences without having to write 
forwarding 

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

adamdebreceni commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r979899914


##
libminifi/include/core/flow/Node.h:
##
@@ -0,0 +1,143 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include "nonstd/expected.hpp"
+
+namespace org::apache::nifi::minifi::core::flow {
+
+class Node {
+ public:
+  struct Cursor {
+int line;
+int column;
+int pos;
+  };
+
+  class Impl;
+  class Iterator {
+   public:
+class Value;
+
+class Impl {
+ public:
+  virtual Impl& operator++() = 0;
+  virtual bool operator==(const Impl& other) const = 0;
+  virtual Value operator*() const = 0;
+  bool operator!=(const Impl& other) const {return !(*this == other);}
+
+  virtual std::unique_ptr clone() const = 0;
+  virtual ~Impl() = default;
+};
+
+Iterator& operator++() {
+  impl_->operator++();
+  return *this;
+}
+
+explicit Iterator(std::unique_ptr impl) : impl_(std::move(impl)) {}
+Iterator(const Iterator& other): impl_(other.impl_->clone()) {}
+Iterator(Iterator&&) = default;
+Iterator& operator=(const Iterator& other) {
+  if (this == &other) {
+return *this;
+  }
+  impl_ = other.impl_->clone();
+  return *this;
+}
+Iterator& operator=(Iterator&&) = default;
+
+bool operator==(const Iterator& other) const {return 
impl_->operator==(*other.impl_);}
+bool operator!=(const Iterator& other) const {return !(*this == other);}
+
+Value operator*() const;
+
+   private:
+std::unique_ptr impl_;
+  };
+
+  class Impl {
+   public:
+virtual explicit operator bool() const = 0;
+virtual bool isSequence() const = 0;
+virtual bool isMap() const = 0;
+virtual bool isNull() const = 0;
+virtual bool isScalar() const = 0;
+
+virtual nonstd::expected getString() 
const = 0;
+virtual nonstd::expected getInt() const = 0;
+virtual nonstd::expected getUInt() const 
= 0;
+virtual nonstd::expected getBool() const = 0;
+virtual nonstd::expected getInt64() const = 0;
+virtual nonstd::expected getUInt64() const = 
0;
+
+virtual std::string getDebugString() const = 0;
+
+virtual size_t size() const = 0;
+virtual Iterator begin() const = 0;
+virtual Iterator end() const = 0;
+virtual Node operator[](std::string_view key) const = 0;
+
+virtual std::optional getCursor() const {return std::nullopt;}
+
+virtual ~Impl() = default;
+  };
+
+  Node() = default;
+  explicit Node(std::shared_ptr impl): impl_(std::move(impl)) {}
+
+  explicit operator bool() const {return impl_->operator bool();}
+  bool isSequence() const {return impl_->isSequence();}
+  bool isMap() const {return impl_->isMap();}
+  bool isNull() const {return impl_->isNull();}
+  bool isScalar() const {return impl_->isScalar();}
+
+  nonstd::expected getString() const {return 
impl_->getString();}
+  nonstd::expected getBool() const {return 
impl_->getBool();}
+  nonstd::expected getInt() const {return 
impl_->getInt();}
+  nonstd::expected getUInt() const {return 
impl_->getUInt();}
+  nonstd::expected getInt64() const {return 
impl_->getInt64();}
+  nonstd::expected getUInt64() const {return 
impl_->getUInt64();}
+
+  // return a string representation of the node (need not to be deserializable)
+  std::string getDebugString() const {return impl_->getDebugString();}
+
+  size_t size() const {return impl_->size();}
+  size_t empty() const {
+return size() == 0;
+  }
+  Iterator begin() const {return impl_->begin();}
+  Iterator end() const {return impl_->end();}
+  Node operator[](std::string_view key) const {return impl_->operator[](key);}
+
+  std::optional getCursor() const {return impl_->getCursor();}
+
+ private:
+  std::shared_ptr impl_;
+};
+
+class Node::Iterator::Value : public Node, public std::pair {
+ public:
+  Value(Node node, Node key, Node value): Node{std::move(node)}, 
std::pair{std::move(key), std::move(value)} {}
+};

Review Comment:
   I wanted to mimic the `yaml-cpp`'s implementation allowing uniform iteration 
on both maps and sequences without having to write forwarding methods if the 
`Node

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

adamdebreceni commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r979885664


##
libminifi/include/core/json/JsonNode.h:
##
@@ -0,0 +1,255 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include "core/flow/Node.h"
+#include "rapidjson/document.h"
+#include "utils/gsl.h"
+
+namespace org::apache::nifi::minifi::core {
+
+class JsonNode : public flow::Node::Impl {
+ public:
+  explicit JsonNode(const rapidjson::Value* node): node_(node) {}
+
+  explicit operator bool() const override {
+return node_ != nullptr;
+  }
+  bool isSequence() const override {
+return node_ ? node_->IsArray() : false;
+  }
+  bool isMap() const override {
+return node_ ? node_->IsObject() : false;
+  }
+  bool isNull() const override {
+return node_ ? node_->IsNull() : false;
+  }
+  bool isScalar() const override {
+// TODO:

Review Comment:
   I was only used in a single place, so I removed this method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

adamdebreceni commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r979885096


##
libminifi/include/core/flow/StructuredConfiguration.h:
##
@@ -0,0 +1,234 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+
+#include "core/FlowConfiguration.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/ProcessorConfig.h"
+#include "Exception.h"
+#include "io/StreamFactory.h"
+#include "io/validation.h"
+#include "sitetosite/SiteToSite.h"
+#include "utils/Id.h"
+#include "utils/StringUtils.h"
+#include "utils/file/FileSystem.h"
+#include "core/flow/Node.h"
+
+namespace org::apache::nifi::minifi::core {

Review Comment:
   makes sense, updated



##
libminifi/include/core/json/JsonConfiguration.h:
##
@@ -0,0 +1,71 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+
+#include "core/FlowConfiguration.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/ProcessorConfig.h"
+#include "Exception.h"
+#include "io/StreamFactory.h"
+#include "io/validation.h"
+#include "sitetosite/SiteToSite.h"
+#include "utils/Id.h"
+#include "utils/StringUtils.h"
+#include "utils/file/FileSystem.h"
+#include "core/flow/StructuredConfiguration.h"
+
+class JsonConfigurationTestAccessor;
+
+namespace org::apache::nifi::minifi::core {
+
+class JsonConfiguration : public StructuredConfiguration {
+ public:
+  explicit JsonConfiguration(ConfigurationContext ctx);
+
+  ~JsonConfiguration() override = default;
+
+  /**
+   * Returns a shared pointer to a ProcessGroup object containing the

Review Comment:
   removed the comments



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

adamdebreceni commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r979884804


##
libminifi/include/core/flow/StructuredConfiguration.h:
##
@@ -0,0 +1,234 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+
+#include "core/FlowConfiguration.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/ProcessorConfig.h"
+#include "Exception.h"
+#include "io/StreamFactory.h"
+#include "io/validation.h"
+#include "sitetosite/SiteToSite.h"
+#include "utils/Id.h"
+#include "utils/StringUtils.h"
+#include "utils/file/FileSystem.h"
+#include "core/flow/Node.h"
+
+namespace org::apache::nifi::minifi::core {
+
+static constexpr char const* CONFIG_FLOW_CONTROLLER_KEY = "Flow Controller";
+static constexpr char const* CONFIG_PROCESSORS_KEY = "Processors";
+static constexpr char const* CONFIG_CONTROLLER_SERVICES_KEY = "Controller 
Services";
+static constexpr char const* CONFIG_REMOTE_PROCESS_GROUP_KEY = "Remote 
Processing Groups";
+static constexpr char const* CONFIG_REMOTE_PROCESS_GROUP_KEY_V3 = "Remote 
Process Groups";
+static constexpr char const* CONFIG_PROVENANCE_REPORT_KEY = "Provenance 
Reporting";
+static constexpr char const* CONFIG_FUNNELS_KEY = "Funnels";
+
+#define CONFIGURATION_USE_REGEX
+
+// Disable regex in EL for incompatible compilers
+#if __GNUC__ < 4 || (__GNUC__ == 4 && __GNUC_MINOR__ < 9)
+#undef CONFIGURATION_USE_REGEX
+#endif

Review Comment:
   removed



##
libminifi/include/core/flow/StructuredConfiguration.h:
##
@@ -0,0 +1,234 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+
+#include "core/FlowConfiguration.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "core/ProcessorConfig.h"
+#include "Exception.h"
+#include "io/StreamFactory.h"
+#include "io/validation.h"
+#include "sitetosite/SiteToSite.h"
+#include "utils/Id.h"
+#include "utils/StringUtils.h"
+#include "utils/file/FileSystem.h"
+#include "core/flow/Node.h"
+
+namespace org::apache::nifi::minifi::core {
+
+static constexpr char const* CONFIG_FLOW_CONTROLLER_KEY = "Flow Controller";
+static constexpr char const* CONFIG_PROCESSORS_KEY = "Processors";
+static constexpr char const* CONFIG_CONTROLLER_SERVICES_KEY = "Controller 
Services";
+static constexpr char const* CONFIG_REMOTE_PROCESS_GROUP_KEY = "Remote 
Processing Groups";
+static constexpr char const* CONFIG_REMOTE_PROCESS_GROUP_KEY_V3 = "Remote 
Process Groups";
+static constexpr char const* CONFIG_PROVENANCE_REPORT_KEY = "Provenance 
Reporting";
+static constexpr char const* CONFIG_FUNNELS_KEY = "Funnels";
+
+#define CONFIGURATION_USE_REGEX
+
+// Disable regex in EL for incompatible compilers
+#if __GNUC__ < 4 || (__GNUC__ == 4 && __GNUC_MINOR__ < 9)
+#undef CONFIGURATION_USE_REGEX
+#endif
+
+class StructuredConfiguration : public FlowConfiguration {
+ public:
+  StructuredConfiguration(ConfigurationContext ctx, 
std::shared_ptr logger);
+
+  /**
+   * Iterates all component property validation rules and checks that 
configured state
+   * is valid. If state is determined to be invalid, conf parsing ends and an 
error is raised.
+   *
+   * @param component
+   * @param component_name
+   * @param section
+   */
+  void validateComponentProperties(ConfigurableComponent& component, const 
std::string &component_name, const std::string §io

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

adamdebreceni commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r979884544


##
libminifi/include/core/flow/README.md:
##
@@ -0,0 +1,57 @@
+## Differences between JSON and YAML implementation
+
+### YAML
+
+The possible types of a `YAML::Node` are:
+* Undefined
+* Null
+* Map
+* Sequence
+* Scalar
+
+ Undefined
+
+The result of querying any member of `Null`, querying non-existing members of 
`Map`,
+or non-existing indices of `Sequence`.
+
+Note that for `Map`s string conversion applies `map[0]` could be valid, given 
a key `"0"`,
+while for `Sequence`s string index parsing does NOT happen `seq["0"]`
+will return `Undefined` even if the sequence is non-empty.
+
+Querying or otherwise accessing an `Undefined` (other than `operator bool` or 
`IsDefined`) usually throws.

Review Comment:
   the `operator bool()` is implemented as `IsDefined()`, so I replaced all 
`IsDefined()` calls with `operator bool()` calls, this part here describes the 
workings of the `YAML::Node`, I did not plan on exposing an `IsDefined`-like 
method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

adamdebreceni commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r979881994


##
libminifi/include/core/flow/Node.h:
##
@@ -0,0 +1,143 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include "nonstd/expected.hpp"
+
+namespace org::apache::nifi::minifi::core::flow {
+
+class Node {
+ public:
+  struct Cursor {
+int line;
+int column;
+int pos;
+  };
+
+  class Impl;
+  class Iterator {
+   public:
+class Value;
+
+class Impl {
+ public:
+  virtual Impl& operator++() = 0;
+  virtual bool operator==(const Impl& other) const = 0;
+  virtual Value operator*() const = 0;
+  bool operator!=(const Impl& other) const {return !(*this == other);}
+
+  virtual std::unique_ptr clone() const = 0;
+  virtual ~Impl() = default;
+};
+
+Iterator& operator++() {
+  impl_->operator++();
+  return *this;
+}
+
+explicit Iterator(std::unique_ptr impl) : impl_(std::move(impl)) {}
+Iterator(const Iterator& other): impl_(other.impl_->clone()) {}
+Iterator(Iterator&&) = default;
+Iterator& operator=(const Iterator& other) {
+  if (this == &other) {
+return *this;
+  }
+  impl_ = other.impl_->clone();
+  return *this;
+}
+Iterator& operator=(Iterator&&) = default;
+
+bool operator==(const Iterator& other) const {return 
impl_->operator==(*other.impl_);}
+bool operator!=(const Iterator& other) const {return !(*this == other);}
+
+Value operator*() const;
+
+   private:
+std::unique_ptr impl_;
+  };
+
+  class Impl {

Review Comment:
   makes sense, renamed them



##
libminifi/include/core/flow/Node.h:
##
@@ -0,0 +1,143 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include "nonstd/expected.hpp"
+
+namespace org::apache::nifi::minifi::core::flow {
+
+class Node {
+ public:
+  struct Cursor {
+int line;
+int column;
+int pos;
+  };
+
+  class Impl;
+  class Iterator {
+   public:
+class Value;
+
+class Impl {
+ public:
+  virtual Impl& operator++() = 0;
+  virtual bool operator==(const Impl& other) const = 0;
+  virtual Value operator*() const = 0;
+  bool operator!=(const Impl& other) const {return !(*this == other);}
+
+  virtual std::unique_ptr clone() const = 0;
+  virtual ~Impl() = default;
+};
+
+Iterator& operator++() {
+  impl_->operator++();
+  return *this;
+}
+
+explicit Iterator(std::unique_ptr impl) : impl_(std::move(impl)) {}
+Iterator(const Iterator& other): impl_(other.impl_->clone()) {}
+Iterator(Iterator&&) = default;
+Iterator& operator=(const Iterator& other) {
+  if (this == &other) {
+return *this;
+  }
+  impl_ = other.impl_->clone();
+  return *this;
+}
+Iterator& operator=(Iterator&&) = default;
+
+bool operator==(const Iterator& other) const {return 
impl_->operator==(*other.impl_);}
+bool operator!=(const Iterator& other) const {return !(*this == other);}
+
+Value operator*() const;
+
+   private:
+std::unique_ptr impl_;
+  };
+
+  class Impl {
+   public:
+virtual explicit operator bool() const = 0;
+virtual bool isSequence() const = 0;
+virtual bool isMap() const = 0;
+virtual bool isNull() const = 0;
+virtual bool is

[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

fgerlits commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r979882035


##
libminifi/include/core/flow/README.md:
##
@@ -0,0 +1,57 @@
+## Differences between JSON and YAML implementation
+
+### YAML
+
+The possible types of a `YAML::Node` are:
+* Undefined
+* Null
+* Map
+* Sequence
+* Scalar
+
+ Undefined
+
+The result of querying any member of `Null`, querying non-existing members of 
`Map`,
+or non-existing indices of `Sequence`.
+
+Note that for `Map`s string conversion applies `map[0]` could be valid, given 
a key `"0"`,
+while for `Sequence`s string index parsing does NOT happen `seq["0"]`
+will return `Undefined` even if the sequence is non-empty.
+
+Querying or otherwise accessing an `Undefined` (other than `operator bool` or 
`IsDefined`) usually throws.
+
+ Null
+
+The value of parsing an empty document, the value of a `Map` item with empty 
value,
+the value of an omitted `Sequence` item.
+
+```
+key1:  # this is a Null
+key2: ''   # this is a Scalar, the empty string
+arr:
+  - one
+  -# Null as well
+  - three
+```
+
+ Scalar
+
+A string value, all conversions to numbers happen on the fly.
+
+### Conversions
+
+ 1. `::as`
+
+* `Null` --> `"null"`
+* `Scalar` --> the string value
+* others --> throws
+
+ 2. `::as`

Review Comment:
   You're right, I hadn't upgraded Clion in a while, and now that I've upgraded 
to 2022.2.3, it is now the other way round :(
   
   But on GitHub itself, `<...>` works and `<...>` doesn't, so I still 
think changing to `<...>` is the right way to go, even if it is going to look 
bad in Clion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1391: MINIFICPP-1846 - Json configuration support part 1

[GitBox]

adamdebreceni commented on code in PR #1391:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1391#discussion_r979881572


##
extensions/standard-processors/tests/unit/YamlConnectionParserTest.cpp:
##
@@ -53,26 +56,21 @@ TEST_CASE("Connections components are parsed from yaml", 
"[YamlConfiguration]")
   "- something_else\n" };
   expectations = { { "success", "" }, { "failure", "" }, { 
"something_else", "" } };
 }
-YAML::Node connection_node = YAML::Load(serialized_yaml);
-YamlConnectionParser yaml_connection_parser(connection_node, "test_node", 
parent_ptr, logger);
-
yaml_connection_parser.configureConnectionSourceRelationshipsFromYaml(*connection);
+YAML::Node yaml_node = YAML::Load(serialized_yaml);
+flow::Node connection_node{std::make_shared(yaml_node)};
+StructuredConnectionParser yaml_connection_parser(connection_node, 
"test_node", parent_ptr, logger);
+yaml_connection_parser.configureConnectionSourceRelationships(*connection);
 const std::set& relationships = 
connection->getRelationships();
 REQUIRE(expectations == relationships);
   }
   SECTION("Queue size limits are read") {
-YAML::Node connection_node = YAML::Load(std::string {
+YAML::Node yaml_node = YAML::Load(std::string {
 "max work queue size: 231\n"
 "max work queue data size: 12 MB\n" });
 YamlConnectionParser yaml_connection_parser(connection_node, "test_node", 
parent_ptr, logger);
 REQUIRE(231 == yaml_connection_parser.getWorkQueueSizeFromYaml());
 REQUIRE(12582912 == 
yaml_connection_parser.getWorkQueueDataSizeFromYaml());  // 12 * 1024 * 1024 B
   }
-  SECTION("Queue swap threshold is read") {
-YAML::Node connection_node = YAML::Load(std::string {
-"swap threshold: 231\n" });
-YamlConnectionParser yaml_connection_parser(connection_node, "test_node", 
parent_ptr, logger);
-REQUIRE(231 == yaml_connection_parser.getSwapThresholdFromYaml());
-  }

Review Comment:
   rebase accident, readded it



##
libminifi/src/core/flow/StructuredConnectionParser.cpp:
##
@@ -108,20 +117,20 @@ uint64_t YamlConnectionParser::getSwapThresholdFromYaml() 
const {
   return 0;

Review Comment:
   changed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request, #1423: MINIFICPP-1941 Upgrade OPC UA library to version 1.3.3

[GitBox]

lordgamez opened a new pull request, #1423:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1423

   https://issues.apache.org/jira/browse/MINIFICPP-1941
   
   
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI 
results for build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request, #1422: MINIFICPP-1938 Enable parallel onTrigger calls for Azure and AWS processors

[GitBox]

lordgamez opened a new pull request, #1422:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1422

   https://issues.apache.org/jira/browse/MINIFICPP-1938
   
   ---
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI 
results for build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org