[jira] [Updated] (NIFI-11409) OIDC Token Revocation Error on Logout
[
https://issues.apache.org/jira/browse/NIFI-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
macdoor615 updated NIFI-11409:
--
Attachment: oracle oauth_revoke_token_flow.png
> OIDC Token Revocation Error on Logout
> -
>
> Key: NIFI-11409
> URL: https://issues.apache.org/jira/browse/NIFI-11409
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.21.0
> Environment: NiFi 1.21.0 cluster with 4 nodes
> openjdk version "11.0.18" 2023-01-17 LTS
> OpenJDK Runtime Environment (Red_Hat-11.0.18.0.10-1.el7_9) (build
> 11.0.18+10-LTS)
> OpenJDK 64-Bit Server VM (Red_Hat-11.0.18.0.10-1.el7_9) (build
> 11.0.18+10-LTS, mixed mode, sharing)
> Linux hb3-ifz-bridge-004 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10
> 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> Keycloak 20.0.2
>Reporter: macdoor615
>Assignee: David Handermann
>Priority: Major
> Attachments: Oracle OAuth 2.0 Flow notes.png, RFC OAuth 2.0 Flow
> notes.png, oracle oauth_revoke_token_flow.png, 截屏2023-04-08 12.40.30.png,
> 截屏2023-04-09 13.17.25.png, 截屏2023-04-09 13.33.25.png
>
>
> My NiFi 1.21.0 cluster has 4 nodes and using oidc authentication.
> I can log in properly, but when I click logout on webui, I got HTTP ERROR 503.
> !截屏2023-04-08 12.40.30.png|width=479,height=179!
> I also find 503 in nifi-request.log
>
> {code:java}
> 10.12.69.33 - - [08/Apr/2023:04:24:13 +] "GET
> /nifi-api/access/oidc/logout HTTP/1.1" 503 425
> "https://36.138.166.203:18088/nifi/; "Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5
> Safari/605.1.15"{code}
>
> and WARNs in nifi-user.log, 36.133.55.100 is load balance's external IP. It
> can not be accessed in intra net.
>
> {code:java}
> 2023-04-08 12:24:43,511 WARN [NiFi Web Server-59]
> o.a.n.w.s.o.r.StandardTokenRevocationResponseClient Token Revocation Request
> processing failed
> org.springframework.web.client.ResourceAccessException: I/O error on POST
> request for
> "https://36.133.55.100:8943/realms/zznode/protocol/openid-connect/revoke":
> connect timed out; nested exception is java.net.SocketTimeoutException:
> connect timed out
> at
> org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:791)
> at
> org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:666)
> at
> org.apache.nifi.web.security.oidc.revocation.StandardTokenRevocationResponseClient.getResponseEntity(StandardTokenRevocationResponseClient.java:81)
> at
> org.apache.nifi.web.security.oidc.revocation.StandardTokenRevocationResponseClient.getRevocationResponse(StandardTokenRevocationResponseClient.java:70)
> at
> org.apache.nifi.web.security.oidc.logout.OidcLogoutSuccessHandler.processRefreshTokenRevocation(OidcLogoutSuccessHandler.java:181)
> at
> org.apache.nifi.web.security.oidc.logout.OidcLogoutSuccessHandler.processLogoutRequest(OidcLogoutSuccessHandler.java:159)
> at
> org.apache.nifi.web.security.oidc.logout.OidcLogoutSuccessHandler.onLogoutSuccess(OidcLogoutSuccessHandler.java:127)
> at
> org.apache.nifi.web.security.logout.StandardLogoutFilter.doFilterInternal(StandardLogoutFilter.java:62)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> at
> org.apache.nifi.web.security.csrf.SkipReplicatedCsrfFilter.doFilterInternal(SkipReplicatedCsrfFilter.java:59)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> at
> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> at
> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:225)
> at
> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190)
> at
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
> at
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
> at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at
>
[jira] [Updated] (NIFI-11409) OIDC Token Revocation Error on Logout
[
https://issues.apache.org/jira/browse/NIFI-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
macdoor615 updated NIFI-11409:
--
Attachment: Oracle OAuth 2.0 Flow notes.png
RFC OAuth 2.0 Flow notes.png
> OIDC Token Revocation Error on Logout
> -
>
> Key: NIFI-11409
> URL: https://issues.apache.org/jira/browse/NIFI-11409
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.21.0
> Environment: NiFi 1.21.0 cluster with 4 nodes
> openjdk version "11.0.18" 2023-01-17 LTS
> OpenJDK Runtime Environment (Red_Hat-11.0.18.0.10-1.el7_9) (build
> 11.0.18+10-LTS)
> OpenJDK 64-Bit Server VM (Red_Hat-11.0.18.0.10-1.el7_9) (build
> 11.0.18+10-LTS, mixed mode, sharing)
> Linux hb3-ifz-bridge-004 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10
> 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> Keycloak 20.0.2
>Reporter: macdoor615
>Assignee: David Handermann
>Priority: Major
> Attachments: Oracle OAuth 2.0 Flow notes.png, RFC OAuth 2.0 Flow
> notes.png, 截屏2023-04-08 12.40.30.png, 截屏2023-04-09 13.17.25.png, 截屏2023-04-09
> 13.33.25.png
>
>
> My NiFi 1.21.0 cluster has 4 nodes and using oidc authentication.
> I can log in properly, but when I click logout on webui, I got HTTP ERROR 503.
> !截屏2023-04-08 12.40.30.png|width=479,height=179!
> I also find 503 in nifi-request.log
>
> {code:java}
> 10.12.69.33 - - [08/Apr/2023:04:24:13 +] "GET
> /nifi-api/access/oidc/logout HTTP/1.1" 503 425
> "https://36.138.166.203:18088/nifi/; "Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5
> Safari/605.1.15"{code}
>
> and WARNs in nifi-user.log, 36.133.55.100 is load balance's external IP. It
> can not be accessed in intra net.
>
> {code:java}
> 2023-04-08 12:24:43,511 WARN [NiFi Web Server-59]
> o.a.n.w.s.o.r.StandardTokenRevocationResponseClient Token Revocation Request
> processing failed
> org.springframework.web.client.ResourceAccessException: I/O error on POST
> request for
> "https://36.133.55.100:8943/realms/zznode/protocol/openid-connect/revoke":
> connect timed out; nested exception is java.net.SocketTimeoutException:
> connect timed out
> at
> org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:791)
> at
> org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:666)
> at
> org.apache.nifi.web.security.oidc.revocation.StandardTokenRevocationResponseClient.getResponseEntity(StandardTokenRevocationResponseClient.java:81)
> at
> org.apache.nifi.web.security.oidc.revocation.StandardTokenRevocationResponseClient.getRevocationResponse(StandardTokenRevocationResponseClient.java:70)
> at
> org.apache.nifi.web.security.oidc.logout.OidcLogoutSuccessHandler.processRefreshTokenRevocation(OidcLogoutSuccessHandler.java:181)
> at
> org.apache.nifi.web.security.oidc.logout.OidcLogoutSuccessHandler.processLogoutRequest(OidcLogoutSuccessHandler.java:159)
> at
> org.apache.nifi.web.security.oidc.logout.OidcLogoutSuccessHandler.onLogoutSuccess(OidcLogoutSuccessHandler.java:127)
> at
> org.apache.nifi.web.security.logout.StandardLogoutFilter.doFilterInternal(StandardLogoutFilter.java:62)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> at
> org.apache.nifi.web.security.csrf.SkipReplicatedCsrfFilter.doFilterInternal(SkipReplicatedCsrfFilter.java:59)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> at
> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> at
> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:225)
> at
> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190)
> at
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
> at
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
> at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at
>
[jira] [Created] (NIFI-11439) GCS processors: add ability to configure custom GCS client endpoint
Paul Grey created NIFI-11439: Summary: GCS processors: add ability to configure custom GCS client endpoint Key: NIFI-11439 URL: https://issues.apache.org/jira/browse/NIFI-11439 Project: Apache NiFi Issue Type: Improvement Reporter: Paul Grey Assignee: Paul Grey AWS processors allow the override of the base endpoint for API calls to AWS services [1]. GCS libraries provide analogous capabilities, as documented here [2]. Expose a GCS processor property to enable GCS endpoint override behavior. [1] https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java#L146-L154 [2] https://cloud.google.com/storage/docs/request-endpoints#storage-set-client-endpoint-java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7117: NIFI-11287: detect dependent properties when the property it depends on references a parameter (UI)
exceptionfactory commented on code in PR #7117:
URL: https://github.com/apache/nifi/pull/7117#discussion_r1164840506
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/jquery/propertytable/jquery.propertytable.js:
##
@@ -2001,6 +2020,45 @@
propertyData.setItems([]);
};
+/**
+ * Gets all the referenced parameters from the {parameterContext} based on
the value of {parameterReference} with matching {sensitive} property
+ *
+ * @param {string} parameterReference
+ * @param {ParameterContextEntity} parameterContext
+ * @param {boolean} sensitive
+ * @returns {ParameterEntity[]}
+ */
+var getExistingParametersReferenced = function (parameterReference,
parameterContext, sensitive) {
Review Comment:
Is there a reason for passing `parameterContext` and `sensitive` as
arguments? It looks like `sensitive` is always `false` and `parameterContext`
always references `currentParameterContext`. Unless there is some other reason
to keep them, it seems like the implementation could be simplified.
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/jquery/propertytable/jquery.propertytable.js:
##
@@ -2261,9 +2319,32 @@
* @argument {object} propertiesThe properties
* @argument {map} descriptors The property descriptors
(property name -> property descriptor)
* @argument {map} history
+ * @argument {object} options
*/
-loadProperties: function (properties, descriptors, history) {
-return this.each(function () {
+loadProperties: function (properties, descriptors, history, options) {
+var self = this;
+var groupId = null;
+
+var loadParameterContext = function (options) {
+if (typeof options.getFullParameterContextDeferred ===
'function') {
Review Comment:
Is there a particular reason for passing `getFullParameterContextDeferred`
and `getParameterContext` as properties of the `options` argument? Could they
be passed as specific arguments instead of properties of `options`? If there is
some other reason, recommend at least providing additional comments for the
`options` argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-11436) FlowUpdateResource throws NPE if PG is not under version control
[
https://issues.apache.org/jira/browse/NIFI-11436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711602#comment-17711602
]
ASF subversion and git services commented on NIFI-11436:
Commit ba93f607315b8a3a43c9fcd0c5fcb1b9668fb701 in nifi's branch
refs/heads/main from Bryan Bende
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ba93f60731 ]
NIFI-11436 Fix NPE during updateFlow when called from a replace request for a
PG that is not under version control (#7167)
> FlowUpdateResource throws NPE if PG is not under version control
>
>
> Key: NIFI-11436
> URL: https://issues.apache.org/jira/browse/NIFI-11436
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 2.0.0, 1.22.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> If the PG being updated is not under version control, then will encounter an
> NPE when trying to retrieve the previous snapshot:
> {code:java}
> java.lang.NullPointerException: null
> at
> org.apache.nifi.web.api.FlowUpdateResource.updateFlow(FlowUpdateResource.java:381)
>
> at
> org.apache.nifi.web.api.FlowUpdateResource.lambda$submitFlowUpdateRequest$5(FlowUpdateResource.java:282)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11436) FlowUpdateResource throws NPE if PG is not under version control
[
https://issues.apache.org/jira/browse/NIFI-11436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711603#comment-17711603
]
ASF subversion and git services commented on NIFI-11436:
Commit 7cb9de69c4c51e233088e0f423d0ceeba03e55b7 in nifi's branch
refs/heads/support/nifi-1.x from Bryan Bende
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7cb9de69c4 ]
NIFI-11436 Fix NPE during updateFlow when called from a replace request for a
PG that is not under version control (#7167)
> FlowUpdateResource throws NPE if PG is not under version control
>
>
> Key: NIFI-11436
> URL: https://issues.apache.org/jira/browse/NIFI-11436
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 2.0.0, 1.22.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> If the PG being updated is not under version control, then will encounter an
> NPE when trying to retrieve the previous snapshot:
> {code:java}
> java.lang.NullPointerException: null
> at
> org.apache.nifi.web.api.FlowUpdateResource.updateFlow(FlowUpdateResource.java:381)
>
> at
> org.apache.nifi.web.api.FlowUpdateResource.lambda$submitFlowUpdateRequest$5(FlowUpdateResource.java:282)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] markap14 merged pull request #7167: NIFI-11436 Fix NPE during updateFlow when called from a replace reque…
markap14 merged PR #7167: URL: https://github.com/apache/nifi/pull/7167 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11436) FlowUpdateResource throws NPE if PG is not under version control
[
https://issues.apache.org/jira/browse/NIFI-11436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-11436:
--
Fix Version/s: 2.0.0
1.22.0
> FlowUpdateResource throws NPE if PG is not under version control
>
>
> Key: NIFI-11436
> URL: https://issues.apache.org/jira/browse/NIFI-11436
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 2.0.0, 1.22.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If the PG being updated is not under version control, then will encounter an
> NPE when trying to retrieve the previous snapshot:
> {code:java}
> java.lang.NullPointerException: null
> at
> org.apache.nifi.web.api.FlowUpdateResource.updateFlow(FlowUpdateResource.java:381)
>
> at
> org.apache.nifi.web.api.FlowUpdateResource.lambda$submitFlowUpdateRequest$5(FlowUpdateResource.java:282)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] markap14 commented on pull request #7167: NIFI-11436 Fix NPE during updateFlow when called from a replace reque…
markap14 commented on PR #7167: URL: https://github.com/apache/nifi/pull/7167#issuecomment-1506089830 Thanks for the fix @bbende . Changes look good +1 will merge to main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11438) OIDC requests all available scopes
[
https://issues.apache.org/jira/browse/NIFI-11438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-11438:
Status: Patch Available (was: In Progress)
> OIDC requests all available scopes
> --
>
> Key: NIFI-11438
> URL: https://issues.apache.org/jira/browse/NIFI-11438
> Project: Apache NiFi
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.21.0
> Environment: Windows ADFS used for OIDC
>Reporter: Jody DesRoches
>Assignee: David Handermann
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> OIDC configuration that works with 1.20.0 fails to login with version 1.21.0.
> Logging exceptions in ADFS that indicate NiFi is requesting forbidden
> resources.
> NiFi is requesting all scopes listed in
> ../adfs/.well-known/openid-configuration under {_}scopes_supported{_}.
> *Expected* only request scopes "{_}openid{_} _email"_ plus values in
> "{_}nifi.security.user.oidc.additional.scopes"{_}
> Source code affecting scope selection:
> [https://github.com/apache/nifi/blob/3322ad7a20c99dec01ee0c3f530c0566acd13258/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProvider.java#L80]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] exceptionfactory opened a new pull request, #7168: NIFI-11438 Set standard OpenID Connect Scopes
exceptionfactory opened a new pull request, #7168: URL: https://github.com/apache/nifi/pull/7168 # Summary [NIFI-11438](https://issues.apache.org/jira/browse/NIFI-11438) Sets the standard requested OpenID Connect scopes to `openid` and `email` to restore the same behavior from NiFi 1.20.0 and earlier. Additional changes include adding the `offline_refresh` scope as the default value in `nifi.properties` to enable Refresh Token requests in the standard configuration. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #4773: NIFI-8161 NiFi EL: migration from SimpleDateFormat to DateTimeFormatter
exceptionfactory commented on PR #4773: URL: https://github.com/apache/nifi/pull/4773#issuecomment-1506047689 It looks like it is not possible to reopen this PR following the rebase. If you can open a new PR, you can reference this one for background. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] arkadius commented on pull request #4773: NIFI-8161 NiFi EL: migration from SimpleDateFormat to DateTimeFormatter
arkadius commented on PR #4773: URL: https://github.com/apache/nifi/pull/4773#issuecomment-1506044009 I rebased it. Can you reopen this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11437) Improve EncryptContentPGP Content Type Detection
[
https://issues.apache.org/jira/browse/NIFI-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-11437:
Affects Version/s: 1.21.0
1.15.0
> Improve EncryptContentPGP Content Type Detection
>
>
> Key: NIFI-11437
> URL: https://issues.apache.org/jira/browse/NIFI-11437
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions, Security
>Affects Versions: 1.15.0, 1.21.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.latest, 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{EncryptContentPGP}} Processor reads the initial bytes of incoming files
> to determine whether a file is an OpenPGP message. This initial read is
> necessary to support flows with {{SignContentPGP}} creating signed OpenPGP
> messages prior to encryption.
> The current implementation uses an InputStreamCallback with
> ProcessSession.read() to run content type detection. Instead a separate read
> callback, the StreamCallback for ProcessSession.write() can be modified to
> use a buffer with {{{}PushbackInputStream{}}}. This will avoid reading the
> initial bytes multiple times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11437) Improve EncryptContentPGP Content Type Detection
[
https://issues.apache.org/jira/browse/NIFI-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-11437:
Issue Type: Bug (was: Improvement)
> Improve EncryptContentPGP Content Type Detection
>
>
> Key: NIFI-11437
> URL: https://issues.apache.org/jira/browse/NIFI-11437
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions, Security
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.latest, 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{EncryptContentPGP}} Processor reads the initial bytes of incoming files
> to determine whether a file is an OpenPGP message. This initial read is
> necessary to support flows with {{SignContentPGP}} creating signed OpenPGP
> messages prior to encryption.
> The current implementation uses an InputStreamCallback with
> ProcessSession.read() to run content type detection. Instead a separate read
> callback, the StreamCallback for ProcessSession.write() can be modified to
> use a buffer with {{{}PushbackInputStream{}}}. This will avoid reading the
> initial bytes multiple times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-11357) Conduct Apache NiFi 1.21.0 release
[ https://issues.apache.org/jira/browse/NIFI-11357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFI-11357. Resolution: Fixed > Conduct Apache NiFi 1.21.0 release > -- > > Key: NIFI-11357 > URL: https://issues.apache.org/jira/browse/NIFI-11357 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Trivial > Fix For: 1.21.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1164569705
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-python-framework/src/main/python/framework/ExtensionManager.py:
##
@@ -0,0 +1,531 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import importlib
+import sys
+import importlib.util # Note requires Python 3.4+
+import inspect
+import logging
+import subprocess
+import ast
+import pkgutil
+from pathlib import Path
+
+logger = logging.getLogger("org.apache.nifi.py4j.ExtensionManager")
+
+# A simple wrapper class to encompass a processor type and its version
+class ExtensionId:
+def __init__(self, classname=None, version=None):
+self.classname = classname
+self.version = version
+
+def __hash__(self):
+return hash((self.classname, self.version))
+
+def __eq__(self, other):
+return (self.classname, self.version) == (other.classname,
other.version)
+
+
+class ExtensionDetails:
+class Java:
+implements = ['org.apache.nifi.python.PythonProcessorDetails']
+
+def __init__(self, gateway, type, version='Unknown', dependencies=None,
source_location=None, package_name=None, description=None, tags=None):
+self.gateway = gateway
+if dependencies is None:
+dependencies = []
+if tags is None:
+tags = []
+
+self.type = type
+self.version = version
+self.dependencies = dependencies
+self.source_location = source_location
+self.package_name = package_name
+self.description = description
+self.tags = tags
+
+def getProcessorType(self):
+return self.type
+
+def getProcessorVersion(self):
+return self.version
+
+def getSourceLocation(self):
+return self.source_location
+
+def getPyPiPackageName(self):
+return self.package_name
+
+def getDependencies(self):
+list = self.gateway.jvm.java.util.ArrayList()
+for dep in self.dependencies:
+list.add(dep)
+
+return list
+
+def getCapabilityDescription(self):
+return self.description
+
+def getTags(self):
+list = self.gateway.jvm.java.util.ArrayList()
+for tag in self.tags:
+list.add(tag)
+
+return list
+
+
+
+
+class ExtensionManager:
+"""
+ExtensionManager is responsible for discovery of extensions types and the
lifecycle management of those extension types.
+Discovery of extension types includes finding what extension types are
available
+(e.g., which Processor types exist on the system), as well as information
about those extension types, such as
+the extension's documentation (tags and capability description).
+
+Lifecycle management includes determining the third-party dependencies
that an extension has and ensuring that those
+third-party dependencies have been imported.
+"""
+
+processorInterfaces =
['org.apache.nifi.python.processor.FlowFileTransform',
'org.apache.nifi.python.processor.RecordTransform']
+processor_details = {}
+processor_class_by_name = {}
+module_files_by_extension_type = {}
+dependency_directories = {}
+
+def __init__(self, gateway):
+self.gateway = gateway
+
+
+def getProcessorTypes(self):
+"""
+:return: a list of Processor types that have been discovered by the
#discoverExtensions method
+"""
+return self.processor_details.values()
+
+def getProcessorClass(self, type, version, work_dir):
+"""
+Returns the Python class that can be used to instantiate a processor
of the given type.
+Additionally, it ensures that the required third-party dependencies
are on the system path in order to ensure that
+the necessary libraries are available to the Processor so that it can
be instantiated and used.
+
+:param type: the type of Processor
+:param version: the version of the Processor
+:param work_dir: the working directory for extensions
+:return: the Python class that can be used to instantiate a Processor
of the given type and version
+
+:raises ValueError: if there
[jira] [Commented] (NIFI-11438) OIDC requests all available scopes
[
https://issues.apache.org/jira/browse/NIFI-11438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711532#comment-17711532
]
David Handermann commented on NIFI-11438:
-
Thanks for the clarification [~dbmxer]. I will work on changing the default
behavior to restore the approach from earlier versions.
> OIDC requests all available scopes
> --
>
> Key: NIFI-11438
> URL: https://issues.apache.org/jira/browse/NIFI-11438
> Project: Apache NiFi
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.21.0
> Environment: Windows ADFS used for OIDC
>Reporter: Jody DesRoches
>Assignee: David Handermann
>Priority: Major
>
> OIDC configuration that works with 1.20.0 fails to login with version 1.21.0.
> Logging exceptions in ADFS that indicate NiFi is requesting forbidden
> resources.
> NiFi is requesting all scopes listed in
> ../adfs/.well-known/openid-configuration under {_}scopes_supported{_}.
> *Expected* only request scopes "{_}openid{_} _email"_ plus values in
> "{_}nifi.security.user.oidc.additional.scopes"{_}
> Source code affecting scope selection:
> [https://github.com/apache/nifi/blob/3322ad7a20c99dec01ee0c3f530c0566acd13258/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProvider.java#L80]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11438) OIDC requests all available scopes
[
https://issues.apache.org/jira/browse/NIFI-11438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711530#comment-17711530
]
Jody DesRoches commented on NIFI-11438:
---
It's possible that the optional scopes listed in the OpenID Connect Core spec
you referenced would be OK. ADFS lists many other scopes that seem very
unrelated to the OpenID workflow used by NiFi. For example, some _*_cert_
values and _user_impersonation_
> OIDC requests all available scopes
> --
>
> Key: NIFI-11438
> URL: https://issues.apache.org/jira/browse/NIFI-11438
> Project: Apache NiFi
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.21.0
> Environment: Windows ADFS used for OIDC
>Reporter: Jody DesRoches
>Assignee: David Handermann
>Priority: Major
>
> OIDC configuration that works with 1.20.0 fails to login with version 1.21.0.
> Logging exceptions in ADFS that indicate NiFi is requesting forbidden
> resources.
> NiFi is requesting all scopes listed in
> ../adfs/.well-known/openid-configuration under {_}scopes_supported{_}.
> *Expected* only request scopes "{_}openid{_} _email"_ plus values in
> "{_}nifi.security.user.oidc.additional.scopes"{_}
> Source code affecting scope selection:
> [https://github.com/apache/nifi/blob/3322ad7a20c99dec01ee0c3f530c0566acd13258/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProvider.java#L80]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11438) OIDC requests all available scopes
[
https://issues.apache.org/jira/browse/NIFI-11438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711528#comment-17711528
]
Jody DesRoches commented on NIFI-11438:
---
[~exceptionfactory] Thanks for jumping on this.
The current behavior in 1.21.0 is to disallow authentication.
> OIDC requests all available scopes
> --
>
> Key: NIFI-11438
> URL: https://issues.apache.org/jira/browse/NIFI-11438
> Project: Apache NiFi
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.21.0
> Environment: Windows ADFS used for OIDC
>Reporter: Jody DesRoches
>Assignee: David Handermann
>Priority: Major
>
> OIDC configuration that works with 1.20.0 fails to login with version 1.21.0.
> Logging exceptions in ADFS that indicate NiFi is requesting forbidden
> resources.
> NiFi is requesting all scopes listed in
> ../adfs/.well-known/openid-configuration under {_}scopes_supported{_}.
> *Expected* only request scopes "{_}openid{_} _email"_ plus values in
> "{_}nifi.security.user.oidc.additional.scopes"{_}
> Source code affecting scope selection:
> [https://github.com/apache/nifi/blob/3322ad7a20c99dec01ee0c3f530c0566acd13258/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProvider.java#L80]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11436) FlowUpdateResource throws NPE if PG is not under version control
[
https://issues.apache.org/jira/browse/NIFI-11436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Bende updated NIFI-11436:
---
Status: Patch Available (was: In Progress)
> FlowUpdateResource throws NPE if PG is not under version control
>
>
> Key: NIFI-11436
> URL: https://issues.apache.org/jira/browse/NIFI-11436
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If the PG being updated is not under version control, then will encounter an
> NPE when trying to retrieve the previous snapshot:
> {code:java}
> java.lang.NullPointerException: null
> at
> org.apache.nifi.web.api.FlowUpdateResource.updateFlow(FlowUpdateResource.java:381)
>
> at
> org.apache.nifi.web.api.FlowUpdateResource.lambda$submitFlowUpdateRequest$5(FlowUpdateResource.java:282)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] bbende opened a new pull request, #7167: NIFI-11436 Fix NPE during updateFlow when called from a replace reque…
bbende opened a new pull request, #7167: URL: https://github.com/apache/nifi/pull/7167 …st for a PG that is not under version control # Summary [NIFI-11436](https://issues.apache.org/jira/browse/NIFI-11436) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [X] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-11438) OIDC requests all available scopes
[
https://issues.apache.org/jira/browse/NIFI-11438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711524#comment-17711524
]
David Handermann commented on NIFI-11438:
-
Thanks for reporting this issue [~dbmxer]. It sounds like changing the behavior
to the previous approach of requesting only {{openid}} and {{email}} may be the
best way forward, although this could also impact Refresh Token retrieval.
The [OpenID Connect Core
specification|https://openid.net/specs/openid-connect-core-1_0.html#ScopeClaims]
defines multiple optional scopes, but does not appear to define particular
behavior when a client requests scopes that the Authorization Server disallows.
For additional background, does the current behavior in NiFi 1.21.0 disallow
authentication altogether, or does it just result in exceptions on AD FS?
> OIDC requests all available scopes
> --
>
> Key: NIFI-11438
> URL: https://issues.apache.org/jira/browse/NIFI-11438
> Project: Apache NiFi
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.21.0
> Environment: Windows ADFS used for OIDC
>Reporter: Jody DesRoches
>Assignee: David Handermann
>Priority: Major
>
> OIDC configuration that works with 1.20.0 fails to login with version 1.21.0.
> Logging exceptions in ADFS that indicate NiFi is requesting forbidden
> resources.
> NiFi is requesting all scopes listed in
> ../adfs/.well-known/openid-configuration under {_}scopes_supported{_}.
> *Expected* only request scopes "{_}openid{_} _email"_ plus values in
> "{_}nifi.security.user.oidc.additional.scopes"{_}
> Source code affecting scope selection:
> [https://github.com/apache/nifi/blob/3322ad7a20c99dec01ee0c3f530c0566acd13258/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProvider.java#L80]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-11438) OIDC requests all available scopes
[
https://issues.apache.org/jira/browse/NIFI-11438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann reassigned NIFI-11438:
---
Assignee: David Handermann
> OIDC requests all available scopes
> --
>
> Key: NIFI-11438
> URL: https://issues.apache.org/jira/browse/NIFI-11438
> Project: Apache NiFi
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.21.0
> Environment: Windows ADFS used for OIDC
>Reporter: Jody DesRoches
>Assignee: David Handermann
>Priority: Major
>
> OIDC configuration that works with 1.20.0 fails to login with version 1.21.0.
> Logging exceptions in ADFS that indicate NiFi is requesting forbidden
> resources.
> NiFi is requesting all scopes listed in
> ../adfs/.well-known/openid-configuration under {_}scopes_supported{_}.
> *Expected* only request scopes "{_}openid{_} _email"_ plus values in
> "{_}nifi.security.user.oidc.additional.scopes"{_}
> Source code affecting scope selection:
> [https://github.com/apache/nifi/blob/3322ad7a20c99dec01ee0c3f530c0566acd13258/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProvider.java#L80]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11438) OIDC requests all available scopes
[
https://issues.apache.org/jira/browse/NIFI-11438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jody DesRoches updated NIFI-11438:
--
Description:
OIDC configuration that works with 1.20.0 fails to login with version 1.21.0.
Logging exceptions in ADFS that indicate NiFi is requesting forbidden resources.
NiFi is requesting all scopes listed in
../adfs/.well-known/openid-configuration under {_}scopes_supported{_}.
*Expected* only request scopes "{_}openid{_} _email"_ plus values in
"{_}nifi.security.user.oidc.additional.scopes"{_}
Source code affecting scope selection:
[https://github.com/apache/nifi/blob/3322ad7a20c99dec01ee0c3f530c0566acd13258/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProvider.java#L80]
was:
OIDC configuration that works with 1.20.0 fails to login with version 1.21.0.
Logging exceptions in ADFS that indicate NiFi is requesting forbidden resources.
NiFi is requesting all scopes listed in
../adfs/.well-known/openid-configuration under {_}scopes_supported{_}.
*Expected* only request scopes "{_}openid{_} ** _email"_ plus values in
"{_}nifi.security.user.oidc.additional.scopes"{_}
Source code affecting scope selection:
https://github.com/apache/nifi/blob/3322ad7a20c99dec01ee0c3f530c0566acd13258/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProvider.java#L80
> OIDC requests all available scopes
> --
>
> Key: NIFI-11438
> URL: https://issues.apache.org/jira/browse/NIFI-11438
> Project: Apache NiFi
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.21.0
> Environment: Windows ADFS used for OIDC
>Reporter: Jody DesRoches
>Priority: Major
>
> OIDC configuration that works with 1.20.0 fails to login with version 1.21.0.
> Logging exceptions in ADFS that indicate NiFi is requesting forbidden
> resources.
> NiFi is requesting all scopes listed in
> ../adfs/.well-known/openid-configuration under {_}scopes_supported{_}.
> *Expected* only request scopes "{_}openid{_} _email"_ plus values in
> "{_}nifi.security.user.oidc.additional.scopes"{_}
> Source code affecting scope selection:
> [https://github.com/apache/nifi/blob/3322ad7a20c99dec01ee0c3f530c0566acd13258/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProvider.java#L80]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-11438) OIDC requests all available scopes
Jody DesRoches created NIFI-11438:
-
Summary: OIDC requests all available scopes
Key: NIFI-11438
URL: https://issues.apache.org/jira/browse/NIFI-11438
Project: Apache NiFi
Issue Type: Bug
Components: Security
Affects Versions: 1.21.0
Environment: Windows ADFS used for OIDC
Reporter: Jody DesRoches
OIDC configuration that works with 1.20.0 fails to login with version 1.21.0.
Logging exceptions in ADFS that indicate NiFi is requesting forbidden resources.
NiFi is requesting all scopes listed in
../adfs/.well-known/openid-configuration under {_}scopes_supported{_}.
*Expected* only request scopes "{_}openid{_} ** _email"_ plus values in
"{_}nifi.security.user.oidc.additional.scopes"{_}
Source code affecting scope selection:
https://github.com/apache/nifi/blob/3322ad7a20c99dec01ee0c3f530c0566acd13258/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/registration/StandardClientRegistrationProvider.java#L80
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on pull request #7160: NIFI-11429 - Upgrade Gremlin to 3.6.2
exceptionfactory commented on PR #7160: URL: https://github.com/apache/nifi/pull/7160#issuecomment-1505730211 Thanks for taking a look @MikeThomsen, I was referring to pull request #5284, which backed down a change from Gremlin 3.5.1 to 3.4.4. The changes apply to the `nifi-other-graph-services` module, so with your background on those services, are you familiar with any potential impact to supported versions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1555: MINIFICPP-2097 Fix build failure when ENABLE_ALL is ON
lordgamez commented on code in PR #1555: URL: https://github.com/apache/nifi-minifi-cpp/pull/1555#discussion_r1164492014 ## extensions/script/CMakeLists.txt: ## Review Comment: I agree, Tensorflow should not be enabled with this flag, it has no bundled libraries and not tested nor built in the CI. If you want to handle the others separately from this issue you can create a separate Jira ticket for it, I'm okay with that too, it's your call. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11437) Improve EncryptContentPGP Content Type Detection
[
https://issues.apache.org/jira/browse/NIFI-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-11437:
Status: Patch Available (was: Open)
> Improve EncryptContentPGP Content Type Detection
>
>
> Key: NIFI-11437
> URL: https://issues.apache.org/jira/browse/NIFI-11437
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions, Security
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.latest, 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{EncryptContentPGP}} Processor reads the initial bytes of incoming files
> to determine whether a file is an OpenPGP message. This initial read is
> necessary to support flows with {{SignContentPGP}} creating signed OpenPGP
> messages prior to encryption.
> The current implementation uses an InputStreamCallback with
> ProcessSession.read() to run content type detection. Instead a separate read
> callback, the StreamCallback for ProcessSession.write() can be modified to
> use a buffer with {{{}PushbackInputStream{}}}. This will avoid reading the
> initial bytes multiple times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] exceptionfactory opened a new pull request, #7166: NIFI-11437 Improve EncryptContentPGP Content Type Detection
exceptionfactory opened a new pull request, #7166: URL: https://github.com/apache/nifi/pull/7166 # Summary [NIFI-11437](https://issues.apache.org/jira/browse/NIFI-11437) Improves content type detection in `EncryptContentPGP` using a [PushbackInputStream](https://docs.oracle.com/javase/8/docs/api/java/io/PushbackInputStream.html) to read up to 8 KB for initial evaluation. The new approach removes the separate call to `ProcessSession.read()` and instead uses a single initial buffer for content type detection. The runtime behavior change is observable in Processor statistics. For a 10 byte file created using `GenerateFlowFile`, the current version of `EncryptContentPGP` would show 20 bytes read. With these changes, `EncryptContentPGP` reports 10 bytes read because the buffered approach avoids multiple reads. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on pull request #7160: NIFI-11429 - Upgrade Gremlin to 3.6.2
MikeThomsen commented on PR #7160: URL: https://github.com/apache/nifi/pull/7160#issuecomment-1505709894 @exceptionfactory Which databases are you talking about? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1555: MINIFICPP-2097 Fix build failure when ENABLE_ALL is ON
fgerlits commented on code in PR #1555: URL: https://github.com/apache/nifi-minifi-cpp/pull/1555#discussion_r1164462363 ## extensions/script/CMakeLists.txt: ## Review Comment: That's a different issue, because they don't cause a build failure, unlike this one. I don't want `ENABLE_ALL` to enable Tensorflow, because that needs special libraries to build, but I can enable the rest of them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-11437) Improve EncryptContentPGP Content Type Detection
David Handermann created NIFI-11437:
---
Summary: Improve EncryptContentPGP Content Type Detection
Key: NIFI-11437
URL: https://issues.apache.org/jira/browse/NIFI-11437
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions, Security
Reporter: David Handermann
Assignee: David Handermann
Fix For: 1.latest, 2.latest
The {{EncryptContentPGP}} Processor reads the initial bytes of incoming files
to determine whether a file is an OpenPGP message. This initial read is
necessary to support flows with {{SignContentPGP}} creating signed OpenPGP
messages prior to encryption.
The current implementation uses an InputStreamCallback with
ProcessSession.read() to run content type detection. Instead a separate read
callback, the StreamCallback for ProcessSession.write() can be modified to use
a buffer with {{{}PushbackInputStream{}}}. This will avoid reading the initial
bytes multiple times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11436) FlowUpdateResource throws NPE if PG is not under version control
[
https://issues.apache.org/jira/browse/NIFI-11436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Bende updated NIFI-11436:
---
Description:
If the PG being updated is not under version control, then will encounter an
NPE when trying to retrieve the previous snapshot:
{code:java}
java.lang.NullPointerException: null
at
org.apache.nifi.web.api.FlowUpdateResource.updateFlow(FlowUpdateResource.java:381)
at
org.apache.nifi.web.api.FlowUpdateResource.lambda$submitFlowUpdateRequest$5(FlowUpdateResource.java:282)
{code}
> FlowUpdateResource throws NPE if PG is not under version control
>
>
> Key: NIFI-11436
> URL: https://issues.apache.org/jira/browse/NIFI-11436
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.21.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
>
> If the PG being updated is not under version control, then will encounter an
> NPE when trying to retrieve the previous snapshot:
> {code:java}
> java.lang.NullPointerException: null
> at
> org.apache.nifi.web.api.FlowUpdateResource.updateFlow(FlowUpdateResource.java:381)
>
> at
> org.apache.nifi.web.api.FlowUpdateResource.lambda$submitFlowUpdateRequest$5(FlowUpdateResource.java:282)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-11436) FlowUpdateResource throws NPE if PG is not under version control
Bryan Bende created NIFI-11436: -- Summary: FlowUpdateResource throws NPE if PG is not under version control Key: NIFI-11436 URL: https://issues.apache.org/jira/browse/NIFI-11436 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.21.0 Reporter: Bryan Bende Assignee: Bryan Bende -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1554: MINIFICPP-2058 Fix AWS extension link error on ARM64
lordgamez commented on code in PR #1554: URL: https://github.com/apache/nifi-minifi-cpp/pull/1554#discussion_r1164391311 ## extensions/aws/CMakeLists.txt: ## Review Comment: It seems that I was wrong as the linkage problem didn't originate from the aws crt library as I originally suspected. All the missing symbols can be resolved if we link the aws-checksums as a whole archive. I tested it on arm and it solves the issue, updated in ec32f02435dc9735994f7c6fb4ab364f34a19266 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1164391462
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/main/java/org/apache/nifi/nar/StandardExtensionDiscoveringManager.java:
##
@@ -169,6 +181,75 @@ public void discoverExtensions(final Set
narBundles, final boolean logDe
}
}
+public void setPythonBridge(final PythonBridge pythonBridge) {
+this.pythonBridge = pythonBridge;
+}
+
+@Override
+public void discoverPythonExtensions(final Bundle pythonBundle) {
+logger.info("Scanning to discover which Python extensions are
available and importing any necessary dependencies. This may take a few
minutes. See python logs for more details.");
+final long start = System.currentTimeMillis();
+pythonBridge.discoverExtensions();
+
+
bundleCoordinateBundleLookup.putIfAbsent(pythonBundle.getBundleDetails().getCoordinate(),
pythonBundle);
+
+final Set processorDefinitions =
definitionMap.get(Processor.class);
+final List pythonProcessorDetails =
pythonBridge.getProcessorTypes();
+
+int processorsFound = 0;
+for (final PythonProcessorDetails details : pythonProcessorDetails) {
+final BundleDetails bundleDetails =
createBundleDetailsWithOverriddenVersion(pythonBundle.getBundleDetails(),
details.getProcessorVersion());
+final Bundle bundle = new Bundle(bundleDetails,
pythonBundle.getClassLoader());
+
+// TODO: This is a workaround because the UI has a bug that causes
it not to work properly if the type doesn't have a '.' in it
+final String className = "python." + details.getProcessorType();
+final ExtensionDefinition extensionDefinition = new
ExtensionDefinition.Builder()
+.implementationClassName(className)
+.runtime(ExtensionRuntime.PYTHON)
+.bundle(bundle)
+.extensionType(Processor.class)
+.description(details.getCapabilityDescription())
+.tags(details.getTags())
+.version(details.getProcessorVersion())
+.build();
+
+final boolean added =
processorDefinitions.add(extensionDefinition);
+if (added) {
+processorsFound++;
+final List bundlesForClass =
classNameBundleLookup.computeIfAbsent(className, key -> new ArrayList<>());
+bundlesForClass.add(bundle);
+
bundleCoordinateBundleLookup.putIfAbsent(bundleDetails.getCoordinate(), bundle);
+logger.info("Discovered Python Processor {}",
details.getProcessorType());
+} else {
+logger.debug("Python Processor {} is already known",
details.getProcessorType());
+}
+}
+
+logger.info("Discovered {} new/updated Python Processors in {}
millis", processorsFound, System.currentTimeMillis() - start);
Review Comment:
I will change it to log at debug level if it doesn't find anything. This
allows us to ensure that the process is happening. But if at least one
processor is found/updated I'll log at info level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1554: MINIFICPP-2058 Fix AWS extension link error on ARM64
lordgamez commented on code in PR #1554: URL: https://github.com/apache/nifi-minifi-cpp/pull/1554#discussion_r1164391311 ## extensions/aws/CMakeLists.txt: ## Review Comment: It seems that I was wrong as the linkage problem didn't originate from the aws crt library as I originally suspected. All the missing symbols can be resolved if we link the aws-checksums as a whole archive. I tested it on arm and it solves the issue, updated in 82e8074988ef5caf6b06e68140c3faae8ce0fd83 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1164388448
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-nar-utils/src/main/java/org/apache/nifi/nar/StandardExtensionDiscoveringManager.java:
##
@@ -169,6 +181,75 @@ public void discoverExtensions(final Set
narBundles, final boolean logDe
}
}
+public void setPythonBridge(final PythonBridge pythonBridge) {
+this.pythonBridge = pythonBridge;
+}
+
+@Override
+public void discoverPythonExtensions(final Bundle pythonBundle) {
+logger.info("Scanning to discover which Python extensions are
available and importing any necessary dependencies. This may take a few
minutes. See python logs for more details.");
Review Comment:
Will add a qualifier to indicate that it may take a few minutes if there
are new extensions discovered. I would avoid mentioning `python.log` explicitly
because it's configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1164383222
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/client/CommandBuilder.java:
##
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.py4j.client;
+
+import py4j.Protocol;
+import py4j.StringUtil;
+import py4j.reflection.ReflectionEngine;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class CommandBuilder {
Review Comment:
Yeah, probably a good call. Will add something here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1164316855
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-python-framework/src/main/python/framework/ExtensionManager.py:
##
@@ -0,0 +1,531 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import importlib
+import sys
+import importlib.util # Note requires Python 3.4+
+import inspect
+import logging
+import subprocess
+import ast
+import pkgutil
+from pathlib import Path
+
+logger = logging.getLogger("org.apache.nifi.py4j.ExtensionManager")
+
+# A simple wrapper class to encompass a processor type and its version
+class ExtensionId:
+def __init__(self, classname=None, version=None):
+self.classname = classname
+self.version = version
+
+def __hash__(self):
+return hash((self.classname, self.version))
+
+def __eq__(self, other):
+return (self.classname, self.version) == (other.classname,
other.version)
+
+
+class ExtensionDetails:
+class Java:
+implements = ['org.apache.nifi.python.PythonProcessorDetails']
+
+def __init__(self, gateway, type, version='Unknown', dependencies=None,
source_location=None, package_name=None, description=None, tags=None):
+self.gateway = gateway
+if dependencies is None:
+dependencies = []
+if tags is None:
+tags = []
+
+self.type = type
+self.version = version
+self.dependencies = dependencies
+self.source_location = source_location
+self.package_name = package_name
+self.description = description
+self.tags = tags
+
+def getProcessorType(self):
+return self.type
+
+def getProcessorVersion(self):
+return self.version
+
+def getSourceLocation(self):
+return self.source_location
+
+def getPyPiPackageName(self):
+return self.package_name
+
+def getDependencies(self):
+list = self.gateway.jvm.java.util.ArrayList()
+for dep in self.dependencies:
+list.add(dep)
+
+return list
+
+def getCapabilityDescription(self):
+return self.description
+
+def getTags(self):
+list = self.gateway.jvm.java.util.ArrayList()
+for tag in self.tags:
+list.add(tag)
+
+return list
+
+
+
+
+class ExtensionManager:
+"""
+ExtensionManager is responsible for discovery of extensions types and the
lifecycle management of those extension types.
+Discovery of extension types includes finding what extension types are
available
+(e.g., which Processor types exist on the system), as well as information
about those extension types, such as
+the extension's documentation (tags and capability description).
+
+Lifecycle management includes determining the third-party dependencies
that an extension has and ensuring that those
+third-party dependencies have been imported.
+"""
+
+processorInterfaces =
['org.apache.nifi.python.processor.FlowFileTransform',
'org.apache.nifi.python.processor.RecordTransform']
+processor_details = {}
+processor_class_by_name = {}
+module_files_by_extension_type = {}
+dependency_directories = {}
+
+def __init__(self, gateway):
+self.gateway = gateway
+
+
+def getProcessorTypes(self):
+"""
+:return: a list of Processor types that have been discovered by the
#discoverExtensions method
+"""
+return self.processor_details.values()
+
+def getProcessorClass(self, type, version, work_dir):
+"""
+Returns the Python class that can be used to instantiate a processor
of the given type.
+Additionally, it ensures that the required third-party dependencies
are on the system path in order to ensure that
+the necessary libraries are available to the Processor so that it can
be instantiated and used.
+
+:param type: the type of Processor
+:param version: the version of the Processor
+:param work_dir: the working directory for extensions
+:return: the Python class that can be used to instantiate a Processor
of the given type and version
+
+:raises ValueError: if there
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1164316855
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-python-framework/src/main/python/framework/ExtensionManager.py:
##
@@ -0,0 +1,531 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import importlib
+import sys
+import importlib.util # Note requires Python 3.4+
+import inspect
+import logging
+import subprocess
+import ast
+import pkgutil
+from pathlib import Path
+
+logger = logging.getLogger("org.apache.nifi.py4j.ExtensionManager")
+
+# A simple wrapper class to encompass a processor type and its version
+class ExtensionId:
+def __init__(self, classname=None, version=None):
+self.classname = classname
+self.version = version
+
+def __hash__(self):
+return hash((self.classname, self.version))
+
+def __eq__(self, other):
+return (self.classname, self.version) == (other.classname,
other.version)
+
+
+class ExtensionDetails:
+class Java:
+implements = ['org.apache.nifi.python.PythonProcessorDetails']
+
+def __init__(self, gateway, type, version='Unknown', dependencies=None,
source_location=None, package_name=None, description=None, tags=None):
+self.gateway = gateway
+if dependencies is None:
+dependencies = []
+if tags is None:
+tags = []
+
+self.type = type
+self.version = version
+self.dependencies = dependencies
+self.source_location = source_location
+self.package_name = package_name
+self.description = description
+self.tags = tags
+
+def getProcessorType(self):
+return self.type
+
+def getProcessorVersion(self):
+return self.version
+
+def getSourceLocation(self):
+return self.source_location
+
+def getPyPiPackageName(self):
+return self.package_name
+
+def getDependencies(self):
+list = self.gateway.jvm.java.util.ArrayList()
+for dep in self.dependencies:
+list.add(dep)
+
+return list
+
+def getCapabilityDescription(self):
+return self.description
+
+def getTags(self):
+list = self.gateway.jvm.java.util.ArrayList()
+for tag in self.tags:
+list.add(tag)
+
+return list
+
+
+
+
+class ExtensionManager:
+"""
+ExtensionManager is responsible for discovery of extensions types and the
lifecycle management of those extension types.
+Discovery of extension types includes finding what extension types are
available
+(e.g., which Processor types exist on the system), as well as information
about those extension types, such as
+the extension's documentation (tags and capability description).
+
+Lifecycle management includes determining the third-party dependencies
that an extension has and ensuring that those
+third-party dependencies have been imported.
+"""
+
+processorInterfaces =
['org.apache.nifi.python.processor.FlowFileTransform',
'org.apache.nifi.python.processor.RecordTransform']
+processor_details = {}
+processor_class_by_name = {}
+module_files_by_extension_type = {}
+dependency_directories = {}
+
+def __init__(self, gateway):
+self.gateway = gateway
+
+
+def getProcessorTypes(self):
+"""
+:return: a list of Processor types that have been discovered by the
#discoverExtensions method
+"""
+return self.processor_details.values()
+
+def getProcessorClass(self, type, version, work_dir):
+"""
+Returns the Python class that can be used to instantiate a processor
of the given type.
+Additionally, it ensures that the required third-party dependencies
are on the system path in order to ensure that
+the necessary libraries are available to the Processor so that it can
be instantiated and used.
+
+:param type: the type of Processor
+:param version: the version of the Processor
+:param work_dir: the working directory for extensions
+:return: the Python class that can be used to instantiate a Processor
of the given type and version
+
+:raises ValueError: if there
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1545: MINIFICPP-55 Implement test coverage report generation
lordgamez commented on code in PR #1545: URL: https://github.com/apache/nifi-minifi-cpp/pull/1545#discussion_r1164200756 ## .github/workflows/ci.yml: ## @@ -156,6 +159,14 @@ jobs: run: cd build && make shellcheck - id: flake8_check run: cd build && make flake8 + - id: coverage Review Comment: I thought it would be useful for the developers to see the test coverage in case of a newly implemented feature, but I suppose that could be done manually on their local environment and the CI would only generate on every push on main. I updated the PR with this condition in ba58ef2cd8438a314eb6bfd5ab25e46335e87be7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink opened a new pull request, #1556: MINIFICPP-2095 fix inconsistent naming in C2 machineArch
martinzink opened a new pull request, #1556: URL: https://github.com/apache/nifi-minifi-cpp/pull/1556 Nifi and minifi java uses machineArch (the other properties also use camelCase), whilst minifi cpp uses machinearch (this makes it hard to create C2 servers that can be used simultaneously with java and cpp agents. --- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file? - [x] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1555: MINIFICPP-2097 Fix build failure when ENABLE_ALL is ON
lordgamez commented on code in PR #1555: URL: https://github.com/apache/nifi-minifi-cpp/pull/1555#discussion_r1164058596 ## extensions/script/CMakeLists.txt: ## Review Comment: I see that there are more extensions where ENABLE_ALL is not take into consideration, should we fix those as well? From what I could gather the extensions where this check is missing are the followings: Bustache, Kubernetes, OPC, OpenCV, OpenWSMAN, PDH, Systemd, Tensorflow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1553: MINIFICPP-2094 Change validators from shared to raw pointers
szaszm commented on code in PR #1553:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1553#discussion_r1163928065
##
libminifi/include/core/CachedValueValidator.h:
##
@@ -67,29 +67,21 @@ class CachedValueValidator {
return *this;
}
- explicit CachedValueValidator(const std::shared_ptr&
other) : validator_(other) {}
+ explicit CachedValueValidator(const gsl::not_null
other) : validator_(other) {}
Review Comment:
I'd change this and `setValidator` to const ref as well, but not the data
member type.
##
libminifi/include/core/Property.h:
##
@@ -96,7 +96,7 @@ class Property {
std::string getDisplayName() const;
std::vector getAllowedTypes() const;
std::string getDescription() const;
- std::shared_ptr getValidator() const;
+ gsl::not_null getValidator() const;
Review Comment:
I would return a `const PropertyValidator&` here, too, for simplicity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits opened a new pull request, #1555: MINIFICPP-2097 Fix build failure when ENABLE_ALL is ON
fgerlits opened a new pull request, #1555: URL: https://github.com/apache/nifi-minifi-cpp/pull/1555 If you try to compile with ENABLE_ALL=ON but with neither ENABLE_PYTHON_SCRIPTING or ENABLE_LUA_SCRIPTING set (so they default to OFF), then the build fails with a "cannot find -lminifi-script-extension: No such file or directory" linker error. This change fixes the error. --- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ChrisSamo632 commented on a diff in pull request #7163: NIFI-11430 - Fixed empty file being produced when grouping by query and fixed _source and _meta extraction by query
ChrisSamo632 commented on code in PR #7163:
URL: https://github.com/apache/nifi/pull/7163#discussion_r1163945827
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractPaginatedJsonQueryElasticsearch.java:
##
@@ -264,14 +265,16 @@ private void combineHits(final List>
hits, final PaginatedJs
* SearchResponse is processed, i.e. this approach allows recursion for
paginated queries, but is unnecessary for single-response queries.
*/
@Override
-List handleHits(final List> hits, final
boolean newQuery, final PaginatedJsonQueryParameters
paginatedJsonQueryParameters,
+List handleHits(List> hits, final boolean
newQuery, final PaginatedJsonQueryParameters paginatedJsonQueryParameters,
final ProcessSession session, final FlowFile
parent, final Map attributes,
final List hitsFlowFiles, final String
transitUri, final StopWatch stopWatch) throws IOException {
paginatedJsonQueryParameters.incrementPageCount();
attributes.put("page.number",
Integer.toString(paginatedJsonQueryParameters.getPageCount()));
if (hitStrategy == ResultOutputStrategy.PER_QUERY) {
-combineHits(hits, paginatedJsonQueryParameters, session, parent,
attributes, hitsFlowFiles);
+
+hits = formatHits(hits);
+combineHits(hits, paginatedJsonQueryParameters, session, parent,
attributes, hitsFlowFiles, newQuery);
Review Comment:
```suggestion
final List> formattedHits = formatHits(hits);
combineHits(formattedHits, paginatedJsonQueryParameters,
session, parent, attributes, hitsFlowFiles, newQuery);
```
Prefer to leave the original input `hits` unchanged for clarity (same as
done in `AbstractJsonQueryElasticsearch#handleHits`)
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/groovy/org/apache/nifi/processors/elasticsearch/AbstractPaginatedJsonQueryElasticsearchTest.groovy:
##
@@ -32,15 +32,15 @@ import static groovy.json.JsonOutput.toJson
import static org.hamcrest.CoreMatchers.equalTo
import static org.hamcrest.CoreMatchers.is
import static org.hamcrest.MatcherAssert.assertThat
-import static org.junit.jupiter.api.Assertions.assertThrows
+import static org.junit.jupiter.api.Assertions.*
Review Comment:
Don't use wildcard imports
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/groovy/org/apache/nifi/processors/elasticsearch/AbstractPaginatedJsonQueryElasticsearchTest.groovy:
##
@@ -117,13 +117,184 @@ abstract class
AbstractPaginatedJsonQueryElasticsearchTest extends AbstractJsonQ
assertSendEvent(runner, input)
}
+@Test
+void testSourceExtraction() throws Exception {
+for (ResultOutputStrategy resultOutputStrategy :
ResultOutputStrategy.values()) {
+final TestRunner runner = createRunner(false)
+runner.setProperty(AbstractJsonQueryElasticsearch.QUERY,
prettyPrint(toJson([query: [match_all: [:]], "sort": [[message: [order:
"asc")))
+
+int flowFileCount
+String hitsCount
+boolean ndjson = false
+
+switch (resultOutputStrategy) {
+case ResultOutputStrategy.PER_QUERY:
+flowFileCount = 1
+hitsCount = "10"
+ndjson = true
+break
+case ResultOutputStrategy.PER_HIT:
+flowFileCount = 10
+hitsCount = "1"
+break
+case ResultOutputStrategy.PER_RESPONSE:
+flowFileCount = 1
+hitsCount = "10"
+break
+}
Review Comment:
Should really have a `default` clause in a `switch` - would likely be a case
of throwing an `IllegalArgumentException` or such here to indicate that we
haven't got a test for the current `resultOutputStrategy` value (e.g. if a new
one is added to the enum in future)
##
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/test/groovy/org/apache/nifi/processors/elasticsearch/AbstractPaginatedJsonQueryElasticsearchTest.groovy:
##
@@ -117,13 +117,184 @@ abstract class
AbstractPaginatedJsonQueryElasticsearchTest extends AbstractJsonQ
assertSendEvent(runner, input)
}
+@Test
+void testSourceExtraction() throws Exception {
+for (ResultOutputStrategy resultOutputStrategy :
ResultOutputStrategy.values()) {
+final TestRunner runner = createRunner(false)
+runner.setProperty(AbstractJsonQueryElasticsearch.QUERY,
prettyPrint(toJson([query: [match_all: [:]], "sort": [[message: [order:
"asc")))
+
+int flowFileCount
+String
[jira] [Created] (MINIFICPP-2097) Fix build failure when ENABLE_ALL is ON but neither script type is explicitly enabled
Ferenc Gerlits created MINIFICPP-2097: - Summary: Fix build failure when ENABLE_ALL is ON but neither script type is explicitly enabled Key: MINIFICPP-2097 URL: https://issues.apache.org/jira/browse/MINIFICPP-2097 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Ferenc Gerlits Assignee: Ferenc Gerlits -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] mr1716 commented on pull request #7162: NIFI-11432 Upgrade Angular Version to 1.8.3
mr1716 commented on PR #7162: URL: https://github.com/apache/nifi/pull/7162#issuecomment-1505128621 All of the checks passed successfully in my local branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1554: MINIFICPP-2058 Fix AWS extension link error on ARM64
szaszm commented on code in PR #1554: URL: https://github.com/apache/nifi-minifi-cpp/pull/1554#discussion_r1163989146 ## extensions/aws/CMakeLists.txt: ## Review Comment: In this case, we should only do the wholearchive linking on the platforms where it's necessary. It generates bloated binaries, and there's no need to do so on x86 if it used to work without this flag there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1554: MINIFICPP-2058 Fix AWS extension link error on ARM64
lordgamez commented on code in PR #1554: URL: https://github.com/apache/nifi-minifi-cpp/pull/1554#discussion_r1163975274 ## extensions/aws/CMakeLists.txt: ## Review Comment: There isn't much documentation on this, most examples I found in other repositories links all the resulting libraries to the client library. It also seemed that not only this single symbol was missing in the linking process, because when this was resolved there were other missing aws symbols from other libraries. Unfortunately this was the only way I could make it work on both ARM64 linux and on Windows as there were other alternative ways that did not work on Windows. ## extensions/aws/CMakeLists.txt: ## Review Comment: There isn't much documentation on this, most examples I found in other repositories linked all the resulting libraries to the client library. It also seemed that not only this single symbol was missing in the linking process, because when this was resolved there were other missing aws symbols from other libraries. Unfortunately this was the only way I could make it work on both ARM64 linux and on Windows as there were other alternative ways that did not work on Windows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1554: MINIFICPP-2058 Fix AWS extension link error on ARM64
szaszm commented on code in PR #1554: URL: https://github.com/apache/nifi-minifi-cpp/pull/1554#discussion_r1163967935 ## extensions/aws/CMakeLists.txt: ## Review Comment: Is there a way to keep that symbol without keeping all of the unused symbols? Or is this the officially documented way? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-2058) AWS extension fails to load on ARM64
[
https://issues.apache.org/jira/browse/MINIFICPP-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gábor Gyimesi updated MINIFICPP-2058:
-
Status: Patch Available (was: Open)
> AWS extension fails to load on ARM64
>
>
> Key: MINIFICPP-2058
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2058
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Gábor Gyimesi
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With the [MINIFICPP-2048 ARM64 support part
> 1|https://github.com/apache/nifi-minifi-cpp/pull/1517] included the AWS
> extension builds but the extension fails to load due to missing symbol
>
> {code:java}
> Failed to load extension 'minifi-aws' ... aws_cpu_has_feature: symbol not
> found
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request, #1554: MINIFICPP-2058 Fix AWS extension link error on ARM64
lordgamez opened a new pull request, #1554: URL: https://github.com/apache/nifi-minifi-cpp/pull/1554 https://issues.apache.org/jira/browse/MINIFICPP-2058 -- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1543: MINIFICPP-2074 Fix time-period/integer validated properties during lo…
szaszm commented on code in PR #1543:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1543#discussion_r1163950530
##
libminifi/src/Configuration.cpp:
##
@@ -179,12 +179,10 @@ std::vector
Configuration::getSensitiveProperties(const std::functi
}
bool Configuration::validatePropertyValue(const std::string& property_name,
const std::string& property_value) {
- for (const auto& config_property: Configuration::CONFIGURATION_PROPERTIES) {
-if (config_property.name == property_name) {
- return config_property.validator->validate(property_name,
property_value).valid();
-}
- }
- return true;
+ if (!Configuration::CONFIGURATION_PROPERTIES.contains(property_name))
+return true;
+
+ return
Configuration::CONFIGURATION_PROPERTIES.at(property_name)->validate(property_name,
property_value).valid();
Review Comment:
We should do the lookup only once, not twice.
```suggestion
const auto validator =
Configuration::CONFIGURATION_PROPERTIES.find(property_name);
if (validator == std::end(Configuration::CONFIGURATION_PROPERTIES))
return true;
return (*validator)->validate(property_name, property_value).valid();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] timeabarna opened a new pull request, #7165: NiFi CLI - add possibility to set 'Maximum Timer Driven Thread Count'
timeabarna opened a new pull request, #7165: URL: https://github.com/apache/nifi/pull/7165 # Summary [NIFI-11435](https://issues.apache.org/jira/browse/NIFI-11435) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] dam4rus commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
dam4rus commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1163915417
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-python-framework/src/main/python/framework/ExtensionManager.py:
##
@@ -0,0 +1,531 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import importlib
+import sys
+import importlib.util # Note requires Python 3.4+
+import inspect
+import logging
+import subprocess
+import ast
+import pkgutil
+from pathlib import Path
+
+logger = logging.getLogger("org.apache.nifi.py4j.ExtensionManager")
+
+# A simple wrapper class to encompass a processor type and its version
+class ExtensionId:
+def __init__(self, classname=None, version=None):
+self.classname = classname
+self.version = version
+
+def __hash__(self):
+return hash((self.classname, self.version))
+
+def __eq__(self, other):
+return (self.classname, self.version) == (other.classname,
other.version)
+
+
+class ExtensionDetails:
+class Java:
+implements = ['org.apache.nifi.python.PythonProcessorDetails']
+
+def __init__(self, gateway, type, version='Unknown', dependencies=None,
source_location=None, package_name=None, description=None, tags=None):
+self.gateway = gateway
+if dependencies is None:
+dependencies = []
+if tags is None:
+tags = []
+
+self.type = type
+self.version = version
+self.dependencies = dependencies
+self.source_location = source_location
+self.package_name = package_name
+self.description = description
+self.tags = tags
+
+def getProcessorType(self):
+return self.type
+
+def getProcessorVersion(self):
+return self.version
+
+def getSourceLocation(self):
+return self.source_location
+
+def getPyPiPackageName(self):
+return self.package_name
+
+def getDependencies(self):
+list = self.gateway.jvm.java.util.ArrayList()
+for dep in self.dependencies:
+list.add(dep)
+
+return list
+
+def getCapabilityDescription(self):
+return self.description
+
+def getTags(self):
+list = self.gateway.jvm.java.util.ArrayList()
+for tag in self.tags:
+list.add(tag)
+
+return list
+
+
+
+
+class ExtensionManager:
+"""
+ExtensionManager is responsible for discovery of extensions types and the
lifecycle management of those extension types.
+Discovery of extension types includes finding what extension types are
available
+(e.g., which Processor types exist on the system), as well as information
about those extension types, such as
+the extension's documentation (tags and capability description).
+
+Lifecycle management includes determining the third-party dependencies
that an extension has and ensuring that those
+third-party dependencies have been imported.
+"""
+
+processorInterfaces =
['org.apache.nifi.python.processor.FlowFileTransform',
'org.apache.nifi.python.processor.RecordTransform']
+processor_details = {}
+processor_class_by_name = {}
+module_files_by_extension_type = {}
+dependency_directories = {}
+
+def __init__(self, gateway):
+self.gateway = gateway
+
+
+def getProcessorTypes(self):
+"""
+:return: a list of Processor types that have been discovered by the
#discoverExtensions method
+"""
+return self.processor_details.values()
+
+def getProcessorClass(self, type, version, work_dir):
+"""
+Returns the Python class that can be used to instantiate a processor
of the given type.
+Additionally, it ensures that the required third-party dependencies
are on the system path in order to ensure that
+the necessary libraries are available to the Processor so that it can
be instantiated and used.
+
+:param type: the type of Processor
+:param version: the version of the Processor
+:param work_dir: the working directory for extensions
+:return: the Python class that can be used to instantiate a Processor
of the given type and version
+
+:raises ValueError: if there
[jira] [Updated] (NIFI-11430) Elasticsearch Paginated Query processor empty hits and extraction issues
[ https://issues.apache.org/jira/browse/NIFI-11430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-11430: -- Status: Patch Available (was: Open) > Elasticsearch Paginated Query processor empty hits and extraction issues > > > Key: NIFI-11430 > URL: https://issues.apache.org/jira/browse/NIFI-11430 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Ryan >Assignee: Ryan >Priority: Major > > If you use the PaginatedJsonQueryProcessor with the following settings > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME > !https://nifi.apache.org/docs/nifi-docs/html/images/iconInfo.png!| > |el-rest-output-no-hits|true| > If you run the processor and hits are returned it outputs 2 flow files, 1 > with the query hits and the other that is empty which is incorrect, it should > return only 1 flow file that contains hits. > > --- > > If you use the PaginatedJsonQueryProcessor with the following settings > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME | > |el-rest-output-no-hits|true| > |el-rest-format-hits|SOURCE_ONLY| > or > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME | > |el-rest-output-no-hits|true| > |el-rest-format-hits|METADATA_ONLY| > If you run the processor and hits are returned it does not extract either > _source or _metadata -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1553: MINIFICPP-2094 Change validators from shared to raw pointers
fgerlits commented on code in PR #1553:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1553#discussion_r1163873063
##
extensions/libarchive/BinFilesStaticDefinitions.cpp:
##
@@ -34,19 +34,19 @@ const core::Property BinFiles::MinSize(
const core::Property BinFiles::MaxSize(
core::PropertyBuilder::createProperty("Maximum Group Size")
->withDescription("The maximum size for the bundle. If not specified,
there is no maximum.")
-
->withType(core::StandardValidators::get().UNSIGNED_LONG_VALIDATOR)->build());
+
->withType(gsl::make_not_null(::StandardValidators::UNSIGNED_LONG_VALIDATOR))->build());
Review Comment:
Yes, that makes the calling site much cleaner. Fixed in
4648366b31e3f1f1d049e0a9a636dc53791105bb.
I have also changed the constructor of `ConfigurationProperty` in
726fa89d2a54c3918d8936a20765c9e6bb46c454, although we may have to undo that
when resolving the conflict with #1543.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] r-vandenbos commented on pull request #7163: Nifi 11430 - Fixed empty file being produced when grouping by query and fixed _source and _meta extraction by query
r-vandenbos commented on PR #7163: URL: https://github.com/apache/nifi/pull/7163#issuecomment-1504954127 @exceptionfactory thanks for pointing this out, I got into a spot a trouble with local vs global settings. This has been fixed now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11435) NiFi CLI - add possibility to set "Maximum Timer Driven Thread Count"
[ https://issues.apache.org/jira/browse/NIFI-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timea Barna updated NIFI-11435: --- Status: Patch Available (was: In Progress) > NiFi CLI - add possibility to set "Maximum Timer Driven Thread Count" > - > > Key: NIFI-11435 > URL: https://issues.apache.org/jira/browse/NIFI-11435 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Timea Barna >Assignee: Timea Barna >Priority: Major > > Add a capability in the NiFi CLI to set the value for "Maximum Timer Driven > Thread Count". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11435) NiFi CLI - add possibility to set "Maximum Timer Driven Thread Count"
Timea Barna created NIFI-11435: -- Summary: NiFi CLI - add possibility to set "Maximum Timer Driven Thread Count" Key: NIFI-11435 URL: https://issues.apache.org/jira/browse/NIFI-11435 Project: Apache NiFi Issue Type: Improvement Reporter: Timea Barna Add a capability in the NiFi CLI to set the value for "Maximum Timer Driven Thread Count". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11435) NiFi CLI - add possibility to set "Maximum Timer Driven Thread Count"
[ https://issues.apache.org/jira/browse/NIFI-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timea Barna reassigned NIFI-11435: -- Assignee: Timea Barna > NiFi CLI - add possibility to set "Maximum Timer Driven Thread Count" > - > > Key: NIFI-11435 > URL: https://issues.apache.org/jira/browse/NIFI-11435 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Timea Barna >Assignee: Timea Barna >Priority: Major > > Add a capability in the NiFi CLI to set the value for "Maximum Timer Driven > Thread Count". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-11430) Elasticsearch Paginated Query processor empty hits and extraction issues
[ https://issues.apache.org/jira/browse/NIFI-11430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan reassigned NIFI-11430: --- Assignee: Ryan > Elasticsearch Paginated Query processor empty hits and extraction issues > > > Key: NIFI-11430 > URL: https://issues.apache.org/jira/browse/NIFI-11430 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Ryan >Assignee: Ryan >Priority: Major > > If you use the PaginatedJsonQueryProcessor with the following settings > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME > !https://nifi.apache.org/docs/nifi-docs/html/images/iconInfo.png!| > |el-rest-output-no-hits|true| > If you run the processor and hits are returned it outputs 2 flow files, 1 > with the query hits and the other that is empty which is incorrect, it should > return only 1 flow file that contains hits. > > --- > > If you use the PaginatedJsonQueryProcessor with the following settings > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME | > |el-rest-output-no-hits|true| > |el-rest-format-hits|SOURCE_ONLY| > or > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME | > |el-rest-output-no-hits|true| > |el-rest-format-hits|METADATA_ONLY| > If you run the processor and hits are returned it does not extract either > _source or _metadata -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2096) Release 0.14.0
[ https://issues.apache.org/jira/browse/MINIFICPP-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gábor Gyimesi updated MINIFICPP-2096: - Fix Version/s: 0.14.0 > Release 0.14.0 > -- > > Key: MINIFICPP-2096 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2096 > Project: Apache NiFi MiNiFi C++ > Issue Type: Task >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Major > Fix For: 0.14.0 > > > Release 0.14.0 of MiNiFi C++. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-2096) Release 0.14.0
Gábor Gyimesi created MINIFICPP-2096: Summary: Release 0.14.0 Key: MINIFICPP-2096 URL: https://issues.apache.org/jira/browse/MINIFICPP-2096 Project: Apache NiFi MiNiFi C++ Issue Type: Task Reporter: Gábor Gyimesi Assignee: Gábor Gyimesi Release 0.14.0 of MiNiFi C++. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2091) Update docker tests with ARM64 supported test images
[ https://issues.apache.org/jira/browse/MINIFICPP-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gábor Gyimesi updated MINIFICPP-2091: - Status: Patch Available (was: Open) > Update docker tests with ARM64 supported test images > > > Key: MINIFICPP-2091 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2091 > Project: Apache NiFi MiNiFi C++ > Issue Type: Test >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We need to be able to run docker system tests on ARM64 to have the same test > coverage as on x86/x64 architectures. There are some docker images used for > testing that are not available with the currently used version on ARM64. > Images that only need a version upgrade to the latest version to support > ARM64: > mcr.microsoft.com/azure-storage/azurite:3.14.2 -> 3.23.0 > apache/nifi:1.16.0 -> 1.20.0 > adobe/s3mock:2.1.28 -> 2.11.0 > apache/nifi-minifi-c2:1.18.0 -> 1.20.0 > These images use the "latest" version should be pinned to a specific version: > fsouza/fake-gcs-server:latest -> 1.44.1 > opensearchproject/opensearch:latest -> 2.6.0 > The following images have no available ARM64 support: > > sameersbn/squid - Used for HTTP proxy, we may be able to find an alternative > proxy image with ARM64 > wurstmeister/zookeeper - Used for kafka tests, we need to find an alternative > zookeeper image > lordgamez/open62541 - Custom built image, as the official repository does not > regularily build their docker image, we need to build a new custom image on > ARM64 arch > splunk/splunk - No officially build image, but can be customly built using > [https://github.com/splunk/docker-splunk] repository according to > https://github.com/splunk/docker-splunk/issues/516 some custom images can > already be found at > [https://hub.docker.com/r/odidev/docker-splunk/tags|https://hub.docker.com/r/odidev/docker-splunk/tags,] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] dam4rus commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
dam4rus commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1163730467
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-python-framework/src/main/python/framework/ExtensionManager.py:
##
@@ -0,0 +1,531 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import importlib
+import sys
+import importlib.util # Note requires Python 3.4+
+import inspect
+import logging
+import subprocess
+import ast
+import pkgutil
+from pathlib import Path
+
+logger = logging.getLogger("org.apache.nifi.py4j.ExtensionManager")
+
+# A simple wrapper class to encompass a processor type and its version
+class ExtensionId:
+def __init__(self, classname=None, version=None):
+self.classname = classname
+self.version = version
+
+def __hash__(self):
+return hash((self.classname, self.version))
+
+def __eq__(self, other):
+return (self.classname, self.version) == (other.classname,
other.version)
+
+
+class ExtensionDetails:
+class Java:
+implements = ['org.apache.nifi.python.PythonProcessorDetails']
+
+def __init__(self, gateway, type, version='Unknown', dependencies=None,
source_location=None, package_name=None, description=None, tags=None):
+self.gateway = gateway
+if dependencies is None:
+dependencies = []
+if tags is None:
+tags = []
+
+self.type = type
+self.version = version
+self.dependencies = dependencies
+self.source_location = source_location
+self.package_name = package_name
+self.description = description
+self.tags = tags
+
+def getProcessorType(self):
+return self.type
+
+def getProcessorVersion(self):
+return self.version
+
+def getSourceLocation(self):
+return self.source_location
+
+def getPyPiPackageName(self):
+return self.package_name
+
+def getDependencies(self):
+list = self.gateway.jvm.java.util.ArrayList()
+for dep in self.dependencies:
+list.add(dep)
+
+return list
+
+def getCapabilityDescription(self):
+return self.description
+
+def getTags(self):
+list = self.gateway.jvm.java.util.ArrayList()
+for tag in self.tags:
+list.add(tag)
+
+return list
+
+
+
+
+class ExtensionManager:
+"""
+ExtensionManager is responsible for discovery of extensions types and the
lifecycle management of those extension types.
+Discovery of extension types includes finding what extension types are
available
+(e.g., which Processor types exist on the system), as well as information
about those extension types, such as
+the extension's documentation (tags and capability description).
+
+Lifecycle management includes determining the third-party dependencies
that an extension has and ensuring that those
+third-party dependencies have been imported.
+"""
+
+processorInterfaces =
['org.apache.nifi.python.processor.FlowFileTransform',
'org.apache.nifi.python.processor.RecordTransform']
+processor_details = {}
+processor_class_by_name = {}
+module_files_by_extension_type = {}
+dependency_directories = {}
+
+def __init__(self, gateway):
+self.gateway = gateway
+
+
+def getProcessorTypes(self):
+"""
+:return: a list of Processor types that have been discovered by the
#discoverExtensions method
+"""
+return self.processor_details.values()
+
+def getProcessorClass(self, type, version, work_dir):
+"""
+Returns the Python class that can be used to instantiate a processor
of the given type.
+Additionally, it ensures that the required third-party dependencies
are on the system path in order to ensure that
+the necessary libraries are available to the Processor so that it can
be instantiated and used.
+
+:param type: the type of Processor
+:param version: the version of the Processor
+:param work_dir: the working directory for extensions
+:return: the Python class that can be used to instantiate a Processor
of the given type and version
+
+:raises ValueError: if there
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1553: MINIFICPP-2094 Change validators from shared to raw pointers
szaszm commented on code in PR #1553:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1553#discussion_r1163707467
##
extensions/libarchive/BinFilesStaticDefinitions.cpp:
##
@@ -34,19 +34,19 @@ const core::Property BinFiles::MinSize(
const core::Property BinFiles::MaxSize(
core::PropertyBuilder::createProperty("Maximum Group Size")
->withDescription("The maximum size for the bundle. If not specified,
there is no maximum.")
-
->withType(core::StandardValidators::get().UNSIGNED_LONG_VALIDATOR)->build());
+
->withType(gsl::make_not_null(::StandardValidators::UNSIGNED_LONG_VALIDATOR))->build());
Review Comment:
I think it would make more sense to have the builder functions take a const
ref to the validators, instead of a not_null pointer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits opened a new pull request, #1553: MINIFICPP-2094 Change validators from shared to raw pointers
fgerlits opened a new pull request, #1553: URL: https://github.com/apache/nifi-minifi-cpp/pull/1553 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
