Re: [PR] NIFI-12785 Refactored InvokeHttp code to avoid double encoding of the entered URL. [nifi]
exceptionfactory commented on code in PR #8458:
URL: https://github.com/apache/nifi/pull/8458#discussion_r1513778626
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java:
##
@@ -1012,7 +1013,7 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
}
}
-private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, URL url) {
+private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, String url) {
Review Comment:
Thanks for the additional explanation. So this should restore the previous
behavior prior to the multiple iterations that resulted from the need to move
away from `java.net.URL` construction.
With this background, it would be helpful to add or update a unit test that
evaluates the Transit URI on a Provenance Event, as one way to confirm the
expected behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12855: Add more information to provenance events to facilitate full graph traversal [nifi]
exceptionfactory commented on PR #8476: URL: https://github.com/apache/nifi/pull/8476#issuecomment-1980030839 I also recommend separating out any GraphClientService changes to their own pull request, as they should be more straightforward to review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12855: Add more information to provenance events to facilitate full graph traversal [nifi]
exceptionfactory commented on PR #8476: URL: https://github.com/apache/nifi/pull/8476#issuecomment-198002 I agree that the FlowFile Repository only has part of the information as well, so an end-to-end solution would require some intermediate layer for sending some metadata elsewhere while maintaining the existing persistence requirements. Taking another look at the options, the `StandardProcessSession` already has access to both transfer relationships and provenance records. At minimum, any changes should not require public-facing `ProvenanceReporter` changes because the `StandardProcessSession` could populate relationship information on provenance records. More evaluation would be required in that direction, but it highlights the importance of scoping any changes as narrowly as possible when it comes to the `nifi-api` interfaces. It might still be necessary to adjust the `ProvenanceEventRecord` definition with this type of approach, but it should not require changing the `ProvenanceEventReporter` and thus all referencing Processors. With this background, it may be best to continue to the discussion on the Jira issue to iron out a general path forward, or providing another different with an alternative solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12855: Add more information to provenance events to facilitate full graph traversal [nifi]
mattyb149 commented on PR #8476: URL: https://github.com/apache/nifi/pull/8476#issuecomment-1979963149 True, I considered that approach but it relies on the FlowFile repository to be as complete as the provenance repository but doesn't it usually "expire" earlier? That's also why I didn't tackle any content claim stuff (although that would be sweet for replay from an external source). Also does it contain relationship information? I didn't see it in the API. I'm totally fine with investigating other approaches, it just seemed nothing quite fit the use cases I was hoping for, mainly to be able to store everything we know in a graph DB for querying / analysis. We can also add the flow graph to the same graph for more powerful queries, so you don't have to do your first search by provenance events, you can find the FlowFile or processor quickly then walk the graph. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12855: Add more information to provenance events to facilitate full graph traversal [nifi]
mattyb149 commented on PR #8476: URL: https://github.com/apache/nifi/pull/8476#issuecomment-1979923174 If a processor issues an event for each FlowFile or for a whole set, they should all go to the relationship. Without having the relationship name you can still link the events but you can't traverse them properly. For example if there's no label on the relationship then when you query for the Flowfiles going through the processors you'll get the events for every path in the tree, instead of being able to query just for "success" for example. These are definitely breaking (API) changes which is why I'm aiming for NiFi 2.0. However if we can't uniquely describe the edge between two provenance events, it will limit future provenance capabilities. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12855) Add more information to provenance events to facilitate full graph traversal
[
https://issues.apache.org/jira/browse/NIFI-12855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-12855:
Status: Patch Available (was: In Progress)
> Add more information to provenance events to facilitate full graph traversal
>
>
> Key: NIFI-12855
> URL: https://issues.apache.org/jira/browse/NIFI-12855
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Although NiFi has a capability in the UI to issue and display lineage queries
> for provenance events, it is not a complete graph that can be traversed if
> for example provenance events were stored in a graph database. The following
> features should be added:
> - A reference in a provenance event to any parent events ("previousEventIds")
> - Add methods to GraphClientService to generate queries/statements in popular
> graph languages such as Tinkerpop/Gremlin, Cypher, and SQL
> - Add explicit references to the relationship to which the FlowFile was
> transferred
> - Add ArcadeDB service as reference implementation for SQL generation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12855: Add more information to provenance events to facilitate full graph traversal [nifi]
mattyb149 opened a new pull request, #8476: URL: https://github.com/apache/nifi/pull/8476 # Summary [NIFI-12855](https://issues.apache.org/jira/browse/NIFI-12855) This PR adds additional information to provenance events (such as previous event IDs) to help facilitate its representation as a property graph (in a graph database for example). # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12630) NPE getLogger in ConsumeSlack
[
https://issues.apache.org/jira/browse/NIFI-12630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823812#comment-17823812
]
ASF subversion and git services commented on NIFI-12630:
Commit bda9b6360d932bb07af387e38a243cb7f3bceb2b in nifi's branch
refs/heads/main from Jim Steinebrey
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=bda9b6360d ]
NIFI-12630 Fix NPE getLogger in ConsumeSlack and PublishSlack
Signed-off-by: Matt Burgess
This closes #8474
> NPE getLogger in ConsumeSlack
> -
>
> Key: NIFI-12630
> URL: https://issues.apache.org/jira/browse/NIFI-12630
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Jim Steinebrey
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Getting this NPE when hitting the Slack rate limit:
> {code:java}
> 2024-01-18 16:07:53,560 WARN [Timer-Driven Process Thread-12]
> o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception
> in Component [ConsumeSlack[id=592d68c7-3fe6-3039-53e4-eae3bfbfbd57]]
> java.lang.NullPointerException: Cannot invoke
> "org.apache.nifi.logging.ComponentLog.debug(String, Object[])" because
> "this.logger" is null
> at
> org.apache.nifi.processors.slack.util.RateLimit.isLimitReached(RateLimit.java:42)
> at
> org.apache.nifi.processors.slack.ConsumeSlack.onTrigger(ConsumeSlack.java:332)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12630) NPE getLogger in ConsumeSlack
[
https://issues.apache.org/jira/browse/NIFI-12630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-12630:
Fix Version/s: 2.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> NPE getLogger in ConsumeSlack
> -
>
> Key: NIFI-12630
> URL: https://issues.apache.org/jira/browse/NIFI-12630
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Jim Steinebrey
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Getting this NPE when hitting the Slack rate limit:
> {code:java}
> 2024-01-18 16:07:53,560 WARN [Timer-Driven Process Thread-12]
> o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception
> in Component [ConsumeSlack[id=592d68c7-3fe6-3039-53e4-eae3bfbfbd57]]
> java.lang.NullPointerException: Cannot invoke
> "org.apache.nifi.logging.ComponentLog.debug(String, Object[])" because
> "this.logger" is null
> at
> org.apache.nifi.processors.slack.util.RateLimit.isLimitReached(RateLimit.java:42)
> at
> org.apache.nifi.processors.slack.ConsumeSlack.onTrigger(ConsumeSlack.java:332)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12630 Fix NPE getLogger in ConsumeSlack and PublishSlack [nifi]
mattyb149 closed pull request #8474: NIFI-12630 Fix NPE getLogger in ConsumeSlack and PublishSlack URL: https://github.com/apache/nifi/pull/8474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12630 Fix NPE getLogger in ConsumeSlack and PublishSlack [nifi]
mattyb149 commented on PR #8474: URL: https://github.com/apache/nifi/pull/8474#issuecomment-1979867767 +1 LGTM, thanks for the fix! Merging to only main until #7673 is backported to support/nifi-1x.x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12630) NPE getLogger in ConsumeSlack
[
https://issues.apache.org/jira/browse/NIFI-12630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-12630:
Affects Version/s: (was: 2.0.0-M1)
(was: 1.24.0)
Status: Patch Available (was: Open)
> NPE getLogger in ConsumeSlack
> -
>
> Key: NIFI-12630
> URL: https://issues.apache.org/jira/browse/NIFI-12630
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Jim Steinebrey
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Getting this NPE when hitting the Slack rate limit:
> {code:java}
> 2024-01-18 16:07:53,560 WARN [Timer-Driven Process Thread-12]
> o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception
> in Component [ConsumeSlack[id=592d68c7-3fe6-3039-53e4-eae3bfbfbd57]]
> java.lang.NullPointerException: Cannot invoke
> "org.apache.nifi.logging.ComponentLog.debug(String, Object[])" because
> "this.logger" is null
> at
> org.apache.nifi.processors.slack.util.RateLimit.isLimitReached(RateLimit.java:42)
> at
> org.apache.nifi.processors.slack.ConsumeSlack.onTrigger(ConsumeSlack.java:332)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12785 Refactored InvokeHttp code to avoid double encoding of the entered URL. [nifi]
dan-s1 commented on code in PR #8458:
URL: https://github.com/apache/nifi/pull/8458#discussion_r1513620936
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java:
##
@@ -1012,7 +1013,7 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
}
}
-private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, URL url) {
+private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, String url) {
Review Comment:
@exceptionfactory Not exactly. Only when creating a URL with the UriUtils
(1.x) / UrlValidator (2.x) is there the double encoding. Before my changes to
how the `java.net.URL` was created there was never a double encoding but if the
`java.net.URL` is created with the `java.net.URI` seven argument constructor it
will always encode "illegal" characters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12785 Refactored InvokeHttp code to avoid double encoding of the entered URL. [nifi]
dan-s1 commented on code in PR #8458:
URL: https://github.com/apache/nifi/pull/8458#discussion_r1513620936
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java:
##
@@ -1012,7 +1013,7 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
}
}
-private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, URL url) {
+private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, String url) {
Review Comment:
@exceptionfactory Not exactly. Only when creating a URL with the UriUtils
(1.x) / UrlValidator (2.x) is there the double encoding. Before my changes to
how the URL was created there was never a double encoding but if the URL is
created with the `java.net.URI` seven argument constructor it will always
encode "illegal" characters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12785 Refactored InvokeHttp code to avoid double encoding of the entered URL. [nifi]
dan-s1 commented on code in PR #8458:
URL: https://github.com/apache/nifi/pull/8458#discussion_r1513620936
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java:
##
@@ -1012,7 +1013,7 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
}
}
-private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, URL url) {
+private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, String url) {
Review Comment:
@exceptionfactory Not exactly. Only when creating a URL with the UriUtils
(v1) / UrlValidator (v2) is there the double encoding. Before my changes to how
the URL was created there was never a double encoding but if the URL is created
with the `java.net.URI` seven argument constructor it will always encode
"illegal" characters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12785 Refactored InvokeHttp code to avoid double encoding of the entered URL. [nifi]
dan-s1 commented on code in PR #8458:
URL: https://github.com/apache/nifi/pull/8458#discussion_r1513620936
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java:
##
@@ -1012,7 +1013,7 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
}
}
-private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, URL url) {
+private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, String url) {
Review Comment:
@exceptionfactory Not exactly. Only when creating a URL with the UriUtils
(v1) / UrlValidator (v2) is there the double encoding. Before my changes to how
the URL was created there was never a double encoding but if the URL is created
with the URI seven argument constructor it will always encode "illegal"
characters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12785 Refactored InvokeHttp code to avoid double encoding of the entered URL. [nifi]
exceptionfactory commented on code in PR #8458:
URL: https://github.com/apache/nifi/pull/8458#discussion_r1513612159
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java:
##
@@ -1012,7 +1013,7 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
}
}
-private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, URL url) {
+private Request configureRequest(final ProcessContext context, final
ProcessSession session, final FlowFile requestFlowFile, String url) {
Review Comment:
For clarification, passing a `URL` results in double-encoding, whereas
passing the raw `String` property does not? How was this working before the
implementation of the custom UriUtils?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12721 Button UX [nifi]
scottyaslan commented on code in PR #8464: URL: https://github.com/apache/nifi/pull/8464#discussion_r1513589282 ## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/assets/themes/nifi.scss: ## @@ -393,7 +393,7 @@ $warn-dark-palette: ( ); // Define the palettes for your theme -$material-primary-light: mat.define-palette($material-primary-light-palette); +$material-primary-light: mat.define-palette($material-primary-light-palette, 600, 100, 900); Review Comment: What is the purpose of this change? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12721 Button UX [nifi]
scottyaslan commented on code in PR #8464:
URL: https://github.com/apache/nifi/pull/8464#discussion_r1513589096
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/styles.scss:
##
@@ -317,7 +317,7 @@ $appFontPath: '~roboto-fontface/fonts';
border-color: $accent-palette-A400;
}
-button:disabled {
+button.nifi-button:disabled {
Review Comment:
Can you please combine this with the style rules directly below this? So
then it would be:
```
button.nifi-button:disabled {
color: $primary-palette-200 !important;
cursor: not-allowed;
border: 1px solid $primary-palette-200;
i {
color: $primary-palette-200 !important;
}
}
```
and there would only be one selector for `button.nifi-button:disabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12720 BUG - Dark mode skeleton loader styles [nifi]
james-elliott commented on code in PR #8465:
URL: https://github.com/apache/nifi/pull/8465#discussion_r1513571432
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/styles.scss:
##
@@ -334,6 +335,9 @@ $appFontPath: '~roboto-fontface/fonts';
.refresh-timestamp {
color: $warn-palette-A400;
}
+.skeleton-loader{
+background: $canvas-primary-palette-500 !important;
Review Comment:
We needed an additional bit of specificity to remove the !important
completely. But otherwise all the comments should be addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-11446) Better handling for cases where Python process dies
[ https://issues.apache.org/jira/browse/NIFI-11446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-11446. --- Fix Version/s: 2.0.0 Assignee: Mark Payne Resolution: Fixed > Better handling for cases where Python process dies > --- > > Key: NIFI-11446 > URL: https://issues.apache.org/jira/browse/NIFI-11446 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 2.0.0 > > > If a Python process dies, we need the ability to detect this, re-launch the > Process, and recreate the Processors that are a part of the Process, and then > restore the Processors' configuration and enable/start them. Essentially, if > the Python process dies, the framework should spawn a new process and allow > the Processor to keep running. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-11444) Improve FlowFileTransform to allow returning a String for the content instead of byte[]
[ https://issues.apache.org/jira/browse/NIFI-11444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-11444. --- Fix Version/s: 2.0.0 Resolution: Fixed > Improve FlowFileTransform to allow returning a String for the content instead > of byte[] > --- > > Key: NIFI-11444 > URL: https://issues.apache.org/jira/browse/NIFI-11444 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Mark Payne >Priority: Major > Fix For: 2.0.0 > > > The FlowFileTransform Python class returns a FlowFileTransformResult. If the > contents are to be returned, they must be provided as a byte[]. But we should > also allow providing the contents as a String and deal with the conversion > behind the scenes, in order to provide a simpler API. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11443) Setup proper logging for Python framework
[ https://issues.apache.org/jira/browse/NIFI-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-11443: -- Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Setup proper logging for Python framework > - > > Key: NIFI-11443 > URL: https://issues.apache.org/jira/browse/NIFI-11443 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Mark Payne >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Currently the python framework establishes logging to logs/nifi-python.log > (directory configured in nifi.properties). But we need to establish proper > logging with log file rotation, etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11443) Setup proper logging for Python framework
[ https://issues.apache.org/jira/browse/NIFI-11443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823784#comment-17823784 ] ASF subversion and git services commented on NIFI-11443: Commit 26f5fa2be044accc6ae8403daf62be621e47f33e in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=26f5fa2be0 ] NIFI-11443 Route Python Framework Logging to SLF4J (#8407) * NIFI-11443 Routed Python Framework Logging to SLF4J - Changed Python logging to use standard output stream - Adjusted Python logging format for simplified processing - Updated PythonProcess to pipe standard error and standard output streams to reader - Added Log Reader command with Virtual Thread for each Python Process - Removed Python log properties from NiFi Properties configuration > Setup proper logging for Python framework > - > > Key: NIFI-11443 > URL: https://issues.apache.org/jira/browse/NIFI-11443 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Mark Payne >Assignee: David Handermann >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Currently the python framework establishes logging to logs/nifi-python.log > (directory configured in nifi.properties). But we need to establish proper > logging with log file rotation, etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]
markap14 merged PR #8407: URL: https://github.com/apache/nifi/pull/8407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]
markap14 commented on PR #8407: URL: https://github.com/apache/nifi/pull/8407#issuecomment-1979701945 Thanks @exceptionfactory all seems to be working well now. This is a huge improvement! +1 will merge to main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12866) DataTypeUtils long to Timestamp conversion may not handle historical dates correctly
Lehel Boér created NIFI-12866:
-
Summary: DataTypeUtils long to Timestamp conversion may not handle
historical dates correctly
Key: NIFI-12866
URL: https://issues.apache.org/jira/browse/NIFI-12866
Project: Apache NiFi
Issue Type: Bug
Reporter: Lehel Boér
Assignee: Lehel Boér
* NiFi version: 1.25.0
* Slack discussion:
https://apachenifi.slack.com/archives/C0L9VCD47/p1709571300887339
When inserting default time -6213559680 (0001-01-01) from NiFi to
PostgreSQL, it results in 0001-01-03.
The conversion happens in DataTypeUtils::toTimeStamp.
{code:java}
if (value instanceof Number) {
final long longValue = ((Number) value).longValue();
return new Timestamp(longValue);
} {code}
When Java created a new Timestamp, it constructs a Date object internally.
When creating a Date object witht he same value it is also shifted.
{code:java}
Date date = new Date(-6213559680L);
1-01-02T19:00:00.000-0500" {code}
This issue needs to be investigated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12862 FlowAnalysisResults should not leak anauthorized component details [nifi]
sardell commented on PR #8475: URL: https://github.com/apache/nifi/pull/8475#issuecomment-1979517205 @tpalfy My mistake. I missed the subjectPermission property in the already-implemented response. Disregard. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513383264
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ControllerServiceResource.java:
##
@@ -680,6 +681,116 @@ public Response updateControllerService(
);
}
+/**
+ * Moves the specified Controller Service to parent/child process groups.
+ *
+ * @param id The id of the controller service to update.
+ * @param requestControllerServiceEntity A controllerServiceEntity.
+ * @return A controllerServiceEntity.
+ */
+@PUT
+@Consumes(MediaType.APPLICATION_JSON)
+@Produces(MediaType.APPLICATION_JSON)
+@Path("{id}/move")
+@Operation(
+summary = "Move Controller Service to the specified Process
Group.",
+responses = @ApiResponse(content = @Content(schema =
@Schema(implementation = ControllerServiceEntity.class))),
+security = {
+@SecurityRequirement(name = "Write -
/controller-services/{uuid}"),
+@SecurityRequirement(name = "Write - Parent Process Group
if scoped by Process Group - /process-groups/{uuid}"),
+@SecurityRequirement(name = "Write - Controller if scoped
by Controller - /controller"),
+@SecurityRequirement(name = "Read - any referenced
Controller Services - /controller-services/{uuid}")
+})
+@ApiResponses(
+value = {
+@ApiResponse(responseCode = "400", description = "NiFi was
unable to complete the request because it was invalid. The request should not
be retried without modification."),
+@ApiResponse(responseCode = "401", description = "Client
could not be authenticated."),
+@ApiResponse(responseCode = "403", description = "Client
is not authorized to make this request."),
+@ApiResponse(responseCode = "404", description = "The
specified resource could not be found."),
+@ApiResponse(responseCode = "409", description = "The
request was valid but NiFi was not in the appropriate state to process it.")
+}
+)
+public Response moveControllerServices(
+@Parameter(
+description = "The controller service id.",
+required = true
+)
+@PathParam("id") final String id,
+@Parameter(
+description = "The controller service entity",
+required = true
+)
+final ControllerServiceEntity requestControllerServiceEntity) {
+
+if (requestControllerServiceEntity == null) {
+throw new IllegalArgumentException("Controller service must be
specified.");
+}
+
+if (requestControllerServiceEntity.getRevision() == null) {
+throw new IllegalArgumentException("Revision must be specified.");
+}
+
+if (requestControllerServiceEntity.getParentGroupId() == null) {
+throw new IllegalArgumentException("ParentGroupId must be
specified.");
+}
+
+final ControllerServiceDTO requestControllerServiceDTO =
serviceFacade.getControllerService(id, true).getComponent();
+ControllerServiceState requestControllerServiceState = null;
+try {
+requestControllerServiceState =
ControllerServiceState.valueOf(requestControllerServiceDTO.getState());
+} catch (final IllegalArgumentException iae) {
+// ignore
+}
+
+// ensure an action has been specified
+if (requestControllerServiceState == null) {
+throw new IllegalArgumentException("Must specify the updated
state. To update the referencing Controller Services the "
++ "state should be DISABLED.");
+}
+
+// ensure the controller service state is not DISABLING
+if
(!ControllerServiceState.DISABLED.equals(requestControllerServiceState) ) {
+throw new IllegalArgumentException("Cannot set the referencing
services to ENABLING or DISABLING");
+}
Review Comment:
Moving the controller service can now be done if it is enabled. removed this
check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513382206
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ControllerServiceResource.java:
##
@@ -680,6 +681,116 @@ public Response updateControllerService(
);
}
+/**
+ * Moves the specified Controller Service to parent/child process groups.
+ *
+ * @param id The id of the controller service to update.
+ * @param requestControllerServiceEntity A controllerServiceEntity.
+ * @return A controllerServiceEntity.
+ */
+@PUT
+@Consumes(MediaType.APPLICATION_JSON)
+@Produces(MediaType.APPLICATION_JSON)
+@Path("{id}/move")
+@Operation(
+summary = "Move Controller Service to the specified Process
Group.",
+responses = @ApiResponse(content = @Content(schema =
@Schema(implementation = ControllerServiceEntity.class))),
+security = {
+@SecurityRequirement(name = "Write -
/controller-services/{uuid}"),
+@SecurityRequirement(name = "Write - Parent Process Group
if scoped by Process Group - /process-groups/{uuid}"),
+@SecurityRequirement(name = "Write - Controller if scoped
by Controller - /controller"),
+@SecurityRequirement(name = "Read - any referenced
Controller Services - /controller-services/{uuid}")
+})
+@ApiResponses(
+value = {
+@ApiResponse(responseCode = "400", description = "NiFi was
unable to complete the request because it was invalid. The request should not
be retried without modification."),
+@ApiResponse(responseCode = "401", description = "Client
could not be authenticated."),
+@ApiResponse(responseCode = "403", description = "Client
is not authorized to make this request."),
+@ApiResponse(responseCode = "404", description = "The
specified resource could not be found."),
+@ApiResponse(responseCode = "409", description = "The
request was valid but NiFi was not in the appropriate state to process it.")
+}
+)
+public Response moveControllerServices(
+@Parameter(
+description = "The controller service id.",
+required = true
+)
+@PathParam("id") final String id,
+@Parameter(
+description = "The controller service entity",
+required = true
+)
+final ControllerServiceEntity requestControllerServiceEntity) {
+
+if (requestControllerServiceEntity == null) {
+throw new IllegalArgumentException("Controller service must be
specified.");
+}
+
+if (requestControllerServiceEntity.getRevision() == null) {
+throw new IllegalArgumentException("Revision must be specified.");
+}
+
+if (requestControllerServiceEntity.getParentGroupId() == null) {
+throw new IllegalArgumentException("ParentGroupId must be
specified.");
+}
+
+final ControllerServiceDTO requestControllerServiceDTO =
serviceFacade.getControllerService(id, true).getComponent();
+ControllerServiceState requestControllerServiceState = null;
+try {
+requestControllerServiceState =
ControllerServiceState.valueOf(requestControllerServiceDTO.getState());
+} catch (final IllegalArgumentException iae) {
+// ignore
+}
+
+// ensure an action has been specified
+if (requestControllerServiceState == null) {
+throw new IllegalArgumentException("Must specify the updated
state. To update the referencing Controller Services the "
++ "state should be DISABLED.");
+}
Review Comment:
Since we no longer care if the controller service is disabled, there were no
longer necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on PR #7734: URL: https://github.com/apache/nifi/pull/7734#issuecomment-1979482152 @markap14 Thank you for the feedback. I've addressed all the changes and concerns provided. The process will now fail if there is a referencing component scope conflict and an error prompt will be shown listing the components that have a scope conflict. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513378795
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-controller-service.js:
##
@@ -2624,6 +2788,26 @@
});
},
+/**
+ * Prompts the user before attempting to move the specified controller
service.
+ *
+ * @param {jQuery} serviceTable
+ * @param {object} controllerServiceEntity
+ */
+promptToMoveController: function (serviceTable,
controllerServiceEntity) {
+// prompt for move
+nfDialog.showYesNoDialog({
+headerText: 'Move Controller Service',
+dialogContent: 'Move controller service \'' +
nfCommon.escapeHtml(controllerServiceEntity.component.name) + '\' '
++ 'to \'' +
nfCommon.escapeHtml($('#move-controller-service-scope').combo('getSelectedOption').text)
+ '\'?'
++ ' '
++ 'Any reference to this controller service by
processors outside the new scope will be removed.',
+yesHandler: function () {
+moveToProcessGroup(serviceTable, controllerServiceEntity,
$('#move-controller-service-scope').combo('getSelectedOption').value);
+}
+});
+},
Review Comment:
Removed prompt. Made it to where the controller service can be moved while
enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513377994
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ControllerServiceResource.java:
##
@@ -680,6 +681,116 @@ public Response updateControllerService(
);
}
+/**
+ * Moves the specified Controller Service to parent/child process groups.
+ *
+ * @param id The id of the controller service to update.
+ * @param requestControllerServiceEntity A controllerServiceEntity.
+ * @return A controllerServiceEntity.
+ */
+@PUT
+@Consumes(MediaType.APPLICATION_JSON)
+@Produces(MediaType.APPLICATION_JSON)
+@Path("{id}/move")
+@Operation(
+summary = "Move Controller Service to the specified Process
Group.",
+responses = @ApiResponse(content = @Content(schema =
@Schema(implementation = ControllerServiceEntity.class))),
+security = {
+@SecurityRequirement(name = "Write -
/controller-services/{uuid}"),
+@SecurityRequirement(name = "Write - Parent Process Group
if scoped by Process Group - /process-groups/{uuid}"),
+@SecurityRequirement(name = "Write - Controller if scoped
by Controller - /controller"),
+@SecurityRequirement(name = "Read - any referenced
Controller Services - /controller-services/{uuid}")
+})
+@ApiResponses(
+value = {
+@ApiResponse(responseCode = "400", description = "NiFi was
unable to complete the request because it was invalid. The request should not
be retried without modification."),
+@ApiResponse(responseCode = "401", description = "Client
could not be authenticated."),
+@ApiResponse(responseCode = "403", description = "Client
is not authorized to make this request."),
+@ApiResponse(responseCode = "404", description = "The
specified resource could not be found."),
+@ApiResponse(responseCode = "409", description = "The
request was valid but NiFi was not in the appropriate state to process it.")
+}
+)
+public Response moveControllerServices(
+@Parameter(
+description = "The controller service id.",
+required = true
+)
+@PathParam("id") final String id,
+@Parameter(
+description = "The controller service entity",
+required = true
+)
+final ControllerServiceEntity requestControllerServiceEntity) {
+
+if (requestControllerServiceEntity == null) {
+throw new IllegalArgumentException("Controller service must be
specified.");
+}
+
+if (requestControllerServiceEntity.getRevision() == null) {
+throw new IllegalArgumentException("Revision must be specified.");
+}
+
+if (requestControllerServiceEntity.getParentGroupId() == null) {
+throw new IllegalArgumentException("ParentGroupId must be
specified.");
+}
+
+final ControllerServiceDTO requestControllerServiceDTO =
serviceFacade.getControllerService(id, true).getComponent();
+ControllerServiceState requestControllerServiceState = null;
+try {
+requestControllerServiceState =
ControllerServiceState.valueOf(requestControllerServiceDTO.getState());
+} catch (final IllegalArgumentException iae) {
+// ignore
+}
+
+// ensure an action has been specified
+if (requestControllerServiceState == null) {
+throw new IllegalArgumentException("Must specify the updated
state. To update the referencing Controller Services the "
++ "state should be DISABLED.");
+}
+
+// ensure the controller service state is not DISABLING
+if
(!ControllerServiceState.DISABLED.equals(requestControllerServiceState) ) {
+throw new IllegalArgumentException("Cannot set the referencing
services to ENABLING or DISABLING");
+}
+
+final Revision requestRevision =
getRevision(requestControllerServiceEntity, id);
+return withWriteLock(
+serviceFacade,
+serviceFacade.getControllerService(id, true),
+requestRevision,
+lookup -> {
+// authorize the service
+final ComponentAuthorizable authorizable =
lookup.getControllerService(requestControllerServiceDTO.getId());
+authorizable.getAuthorizable().authorize(authorizer,
RequestAction.WRITE, NiFiUserUtils.getNiFiUser());
+
+// authorize the current and new process groups
+final ProcessGroupAuthorizable authorizableProcessGroupNew
=
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513377110
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##
@@ -2939,6 +2939,115 @@ public ControllerServiceEntity
updateControllerService(final Revision revision,
return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
}
+@Override
+public ControllerServiceEntity moveControllerService(final Revision
revision, final ControllerServiceDTO controllerServiceDTO, final String
newProcessGroupID) {
+// get the component, ensure we have access to it, and perform the
move request
+final ControllerServiceNode controllerService =
controllerServiceDAO.getControllerService(controllerServiceDTO.getId());
+final RevisionUpdate snapshot =
updateComponent(revision,
+controllerService,
+() -> moveControllerServiceWork(controllerService,
newProcessGroupID),
+cs -> {
+awaitValidationCompletion(cs);
+final ControllerServiceDTO dto =
dtoFactory.createControllerServiceDto(cs);
+final ControllerServiceReference ref =
controllerService.getReferences();
+final ControllerServiceReferencingComponentsEntity
referencingComponentsEntity =
createControllerServiceReferencingComponentsEntity(ref);
+
dto.setReferencingComponents(referencingComponentsEntity.getControllerServiceReferencingComponents());
+return dto;
+});
+
+final PermissionsDTO permissions =
dtoFactory.createPermissionsDto(controllerService);
+final PermissionsDTO operatePermissions =
dtoFactory.createPermissionsDto(new OperationAuthorizable(controllerService));
+final List bulletins =
dtoFactory.createBulletinDtos(bulletinRepository.findBulletinsForSource(controllerServiceDTO.getId()));
+final List bulletinEntities =
bulletins.stream().map(bulletin -> entityFactory.createBulletinEntity(bulletin,
permissions.getCanRead())).collect(Collectors.toList());
+return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
+}
+
+private ControllerServiceNode moveControllerServiceWork(final
ControllerServiceNode controllerService, final String newProcessGroupID) {
+final ProcessGroup oldParentGroup =
controllerService.getProcessGroup();
+Set referencedComponents =
controllerService.getReferences().getReferencingComponents();
+isReferencesDisabled(referencedComponents);
+oldParentGroup.removeControllerService(controllerService);
+ProcessGroup newParent;
+if (!oldParentGroup.isRootGroup() &&
oldParentGroup.getParent().getIdentifier().equals(newProcessGroupID)) {
+// move to parent process group
+newParent = oldParentGroup.getParent();
+newParent.addControllerService(controllerService);
+
+// unset any references the controller services has to other
controller services that are now out of scope
+Map updatedProps = new HashMap<>();
+Set>
properties = controllerService.getProperties().entrySet();
+for (var prop : properties) {
+var value = prop.getValue();
+if (value !=null) {
+ControllerServiceNode controller;
+try {
+controller =
controllerServiceDAO.getControllerService((value.getRawValue()));
+} catch (Exception e){
+continue;
+}
+if (controller != null) {
+if (!hasProcessGroup(controller.getProcessGroup(),
newParent.getIdentifier())) {
+controller.removeReference(controllerService,
prop.getKey());
+updatedProps.put(prop.getKey().getName(), null);
+}
+}
+}
+}
+if (!updatedProps.isEmpty())
+controllerService.setProperties(updatedProps, true,
Collections.emptySet());
+
+} else {
+// move to child process group
+newParent = oldParentGroup.getProcessGroup(newProcessGroupID);
+newParent.addControllerService(controllerService);
+
+// unset any references for processors that are outside the new
scope
+for (ComponentNode node : referencedComponents) {
+if (!hasProcessGroup(newParent,
node.getProcessGroupIdentifier())) {
+Set>
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513376790
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##
@@ -2939,6 +2939,115 @@ public ControllerServiceEntity
updateControllerService(final Revision revision,
return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
}
+@Override
+public ControllerServiceEntity moveControllerService(final Revision
revision, final ControllerServiceDTO controllerServiceDTO, final String
newProcessGroupID) {
+// get the component, ensure we have access to it, and perform the
move request
+final ControllerServiceNode controllerService =
controllerServiceDAO.getControllerService(controllerServiceDTO.getId());
+final RevisionUpdate snapshot =
updateComponent(revision,
+controllerService,
+() -> moveControllerServiceWork(controllerService,
newProcessGroupID),
+cs -> {
+awaitValidationCompletion(cs);
+final ControllerServiceDTO dto =
dtoFactory.createControllerServiceDto(cs);
+final ControllerServiceReference ref =
controllerService.getReferences();
+final ControllerServiceReferencingComponentsEntity
referencingComponentsEntity =
createControllerServiceReferencingComponentsEntity(ref);
+
dto.setReferencingComponents(referencingComponentsEntity.getControllerServiceReferencingComponents());
+return dto;
+});
+
+final PermissionsDTO permissions =
dtoFactory.createPermissionsDto(controllerService);
+final PermissionsDTO operatePermissions =
dtoFactory.createPermissionsDto(new OperationAuthorizable(controllerService));
+final List bulletins =
dtoFactory.createBulletinDtos(bulletinRepository.findBulletinsForSource(controllerServiceDTO.getId()));
+final List bulletinEntities =
bulletins.stream().map(bulletin -> entityFactory.createBulletinEntity(bulletin,
permissions.getCanRead())).collect(Collectors.toList());
+return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
+}
+
+private ControllerServiceNode moveControllerServiceWork(final
ControllerServiceNode controllerService, final String newProcessGroupID) {
+final ProcessGroup oldParentGroup =
controllerService.getProcessGroup();
+Set referencedComponents =
controllerService.getReferences().getReferencingComponents();
+isReferencesDisabled(referencedComponents);
+oldParentGroup.removeControllerService(controllerService);
+ProcessGroup newParent;
+if (!oldParentGroup.isRootGroup() &&
oldParentGroup.getParent().getIdentifier().equals(newProcessGroupID)) {
+// move to parent process group
+newParent = oldParentGroup.getParent();
+newParent.addControllerService(controllerService);
+
+// unset any references the controller services has to other
controller services that are now out of scope
+Map updatedProps = new HashMap<>();
+Set>
properties = controllerService.getProperties().entrySet();
+for (var prop : properties) {
+var value = prop.getValue();
+if (value !=null) {
+ControllerServiceNode controller;
+try {
+controller =
controllerServiceDAO.getControllerService((value.getRawValue()));
+} catch (Exception e){
+continue;
+}
+if (controller != null) {
+if (!hasProcessGroup(controller.getProcessGroup(),
newParent.getIdentifier())) {
+controller.removeReference(controllerService,
prop.getKey());
Review Comment:
The move process now fails and shows an error message when there is a scope
conflict with a referencing component. References are no longer removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513375471
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##
@@ -2939,6 +2939,115 @@ public ControllerServiceEntity
updateControllerService(final Revision revision,
return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
}
+@Override
+public ControllerServiceEntity moveControllerService(final Revision
revision, final ControllerServiceDTO controllerServiceDTO, final String
newProcessGroupID) {
+// get the component, ensure we have access to it, and perform the
move request
+final ControllerServiceNode controllerService =
controllerServiceDAO.getControllerService(controllerServiceDTO.getId());
+final RevisionUpdate snapshot =
updateComponent(revision,
+controllerService,
+() -> moveControllerServiceWork(controllerService,
newProcessGroupID),
+cs -> {
+awaitValidationCompletion(cs);
+final ControllerServiceDTO dto =
dtoFactory.createControllerServiceDto(cs);
+final ControllerServiceReference ref =
controllerService.getReferences();
+final ControllerServiceReferencingComponentsEntity
referencingComponentsEntity =
createControllerServiceReferencingComponentsEntity(ref);
+
dto.setReferencingComponents(referencingComponentsEntity.getControllerServiceReferencingComponents());
+return dto;
+});
+
+final PermissionsDTO permissions =
dtoFactory.createPermissionsDto(controllerService);
+final PermissionsDTO operatePermissions =
dtoFactory.createPermissionsDto(new OperationAuthorizable(controllerService));
+final List bulletins =
dtoFactory.createBulletinDtos(bulletinRepository.findBulletinsForSource(controllerServiceDTO.getId()));
+final List bulletinEntities =
bulletins.stream().map(bulletin -> entityFactory.createBulletinEntity(bulletin,
permissions.getCanRead())).collect(Collectors.toList());
+return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
+}
+
+private ControllerServiceNode moveControllerServiceWork(final
ControllerServiceNode controllerService, final String newProcessGroupID) {
+final ProcessGroup oldParentGroup =
controllerService.getProcessGroup();
+Set referencedComponents =
controllerService.getReferences().getReferencingComponents();
+isReferencesDisabled(referencedComponents);
+oldParentGroup.removeControllerService(controllerService);
+ProcessGroup newParent;
+if (!oldParentGroup.isRootGroup() &&
oldParentGroup.getParent().getIdentifier().equals(newProcessGroupID)) {
+// move to parent process group
+newParent = oldParentGroup.getParent();
+newParent.addControllerService(controllerService);
+
+// unset any references the controller services has to other
controller services that are now out of scope
+Map updatedProps = new HashMap<>();
+Set>
properties = controllerService.getProperties().entrySet();
+for (var prop : properties) {
+var value = prop.getValue();
+if (value !=null) {
+ControllerServiceNode controller;
+try {
+controller =
controllerServiceDAO.getControllerService((value.getRawValue()));
+} catch (Exception e){
+continue;
+}
+if (controller != null) {
Review Comment:
Fixed, but then removed as it was no longer needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513374784
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##
@@ -2939,6 +2939,115 @@ public ControllerServiceEntity
updateControllerService(final Revision revision,
return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
}
+@Override
+public ControllerServiceEntity moveControllerService(final Revision
revision, final ControllerServiceDTO controllerServiceDTO, final String
newProcessGroupID) {
+// get the component, ensure we have access to it, and perform the
move request
+final ControllerServiceNode controllerService =
controllerServiceDAO.getControllerService(controllerServiceDTO.getId());
+final RevisionUpdate snapshot =
updateComponent(revision,
+controllerService,
+() -> moveControllerServiceWork(controllerService,
newProcessGroupID),
+cs -> {
+awaitValidationCompletion(cs);
+final ControllerServiceDTO dto =
dtoFactory.createControllerServiceDto(cs);
+final ControllerServiceReference ref =
controllerService.getReferences();
+final ControllerServiceReferencingComponentsEntity
referencingComponentsEntity =
createControllerServiceReferencingComponentsEntity(ref);
+
dto.setReferencingComponents(referencingComponentsEntity.getControllerServiceReferencingComponents());
+return dto;
+});
+
+final PermissionsDTO permissions =
dtoFactory.createPermissionsDto(controllerService);
+final PermissionsDTO operatePermissions =
dtoFactory.createPermissionsDto(new OperationAuthorizable(controllerService));
+final List bulletins =
dtoFactory.createBulletinDtos(bulletinRepository.findBulletinsForSource(controllerServiceDTO.getId()));
+final List bulletinEntities =
bulletins.stream().map(bulletin -> entityFactory.createBulletinEntity(bulletin,
permissions.getCanRead())).collect(Collectors.toList());
+return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
+}
+
+private ControllerServiceNode moveControllerServiceWork(final
ControllerServiceNode controllerService, final String newProcessGroupID) {
+final ProcessGroup oldParentGroup =
controllerService.getProcessGroup();
+Set referencedComponents =
controllerService.getReferences().getReferencingComponents();
+isReferencesDisabled(referencedComponents);
+oldParentGroup.removeControllerService(controllerService);
+ProcessGroup newParent;
+if (!oldParentGroup.isRootGroup() &&
oldParentGroup.getParent().getIdentifier().equals(newProcessGroupID)) {
+// move to parent process group
+newParent = oldParentGroup.getParent();
+newParent.addControllerService(controllerService);
+
+// unset any references the controller services has to other
controller services that are now out of scope
+Map updatedProps = new HashMap<>();
+Set>
properties = controllerService.getProperties().entrySet();
+for (var prop : properties) {
+var value = prop.getValue();
+if (value !=null) {
+ControllerServiceNode controller;
+try {
+controller =
controllerServiceDAO.getControllerService((value.getRawValue()));
+} catch (Exception e){
+continue;
+}
Review Comment:
Fixed, but then removed as it was no longer needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513374477
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##
@@ -2939,6 +2939,115 @@ public ControllerServiceEntity
updateControllerService(final Revision revision,
return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
}
+@Override
+public ControllerServiceEntity moveControllerService(final Revision
revision, final ControllerServiceDTO controllerServiceDTO, final String
newProcessGroupID) {
+// get the component, ensure we have access to it, and perform the
move request
+final ControllerServiceNode controllerService =
controllerServiceDAO.getControllerService(controllerServiceDTO.getId());
+final RevisionUpdate snapshot =
updateComponent(revision,
+controllerService,
+() -> moveControllerServiceWork(controllerService,
newProcessGroupID),
+cs -> {
+awaitValidationCompletion(cs);
+final ControllerServiceDTO dto =
dtoFactory.createControllerServiceDto(cs);
+final ControllerServiceReference ref =
controllerService.getReferences();
+final ControllerServiceReferencingComponentsEntity
referencingComponentsEntity =
createControllerServiceReferencingComponentsEntity(ref);
+
dto.setReferencingComponents(referencingComponentsEntity.getControllerServiceReferencingComponents());
+return dto;
+});
+
+final PermissionsDTO permissions =
dtoFactory.createPermissionsDto(controllerService);
+final PermissionsDTO operatePermissions =
dtoFactory.createPermissionsDto(new OperationAuthorizable(controllerService));
+final List bulletins =
dtoFactory.createBulletinDtos(bulletinRepository.findBulletinsForSource(controllerServiceDTO.getId()));
+final List bulletinEntities =
bulletins.stream().map(bulletin -> entityFactory.createBulletinEntity(bulletin,
permissions.getCanRead())).collect(Collectors.toList());
+return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
+}
+
+private ControllerServiceNode moveControllerServiceWork(final
ControllerServiceNode controllerService, final String newProcessGroupID) {
+final ProcessGroup oldParentGroup =
controllerService.getProcessGroup();
+Set referencedComponents =
controllerService.getReferences().getReferencingComponents();
+isReferencesDisabled(referencedComponents);
+oldParentGroup.removeControllerService(controllerService);
+ProcessGroup newParent;
+if (!oldParentGroup.isRootGroup() &&
oldParentGroup.getParent().getIdentifier().equals(newProcessGroupID)) {
+// move to parent process group
+newParent = oldParentGroup.getParent();
+newParent.addControllerService(controllerService);
+
+// unset any references the controller services has to other
controller services that are now out of scope
+Map updatedProps = new HashMap<>();
+Set>
properties = controllerService.getProperties().entrySet();
+for (var prop : properties) {
Review Comment:
Removed usages of var
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513374037
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##
@@ -2939,6 +2939,115 @@ public ControllerServiceEntity
updateControllerService(final Revision revision,
return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
}
+@Override
+public ControllerServiceEntity moveControllerService(final Revision
revision, final ControllerServiceDTO controllerServiceDTO, final String
newProcessGroupID) {
+// get the component, ensure we have access to it, and perform the
move request
+final ControllerServiceNode controllerService =
controllerServiceDAO.getControllerService(controllerServiceDTO.getId());
+final RevisionUpdate snapshot =
updateComponent(revision,
+controllerService,
+() -> moveControllerServiceWork(controllerService,
newProcessGroupID),
+cs -> {
+awaitValidationCompletion(cs);
+final ControllerServiceDTO dto =
dtoFactory.createControllerServiceDto(cs);
+final ControllerServiceReference ref =
controllerService.getReferences();
+final ControllerServiceReferencingComponentsEntity
referencingComponentsEntity =
createControllerServiceReferencingComponentsEntity(ref);
+
dto.setReferencingComponents(referencingComponentsEntity.getControllerServiceReferencingComponents());
+return dto;
+});
+
+final PermissionsDTO permissions =
dtoFactory.createPermissionsDto(controllerService);
+final PermissionsDTO operatePermissions =
dtoFactory.createPermissionsDto(new OperationAuthorizable(controllerService));
+final List bulletins =
dtoFactory.createBulletinDtos(bulletinRepository.findBulletinsForSource(controllerServiceDTO.getId()));
+final List bulletinEntities =
bulletins.stream().map(bulletin -> entityFactory.createBulletinEntity(bulletin,
permissions.getCanRead())).collect(Collectors.toList());
+return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
+}
+
+private ControllerServiceNode moveControllerServiceWork(final
ControllerServiceNode controllerService, final String newProcessGroupID) {
+final ProcessGroup oldParentGroup =
controllerService.getProcessGroup();
+Set referencedComponents =
controllerService.getReferences().getReferencingComponents();
+isReferencesDisabled(referencedComponents);
+oldParentGroup.removeControllerService(controllerService);
+ProcessGroup newParent;
+if (!oldParentGroup.isRootGroup() &&
oldParentGroup.getParent().getIdentifier().equals(newProcessGroupID)) {
+// move to parent process group
+newParent = oldParentGroup.getParent();
+newParent.addControllerService(controllerService);
+
+// unset any references the controller services has to other
controller services that are now out of scope
+Map updatedProps = new HashMap<>();
+Set>
properties = controllerService.getProperties().entrySet();
+for (var prop : properties) {
+var value = prop.getValue();
+if (value !=null) {
+ControllerServiceNode controller;
+try {
+controller =
controllerServiceDAO.getControllerService((value.getRawValue()));
+} catch (Exception e){
+continue;
+}
+if (controller != null) {
+if (!hasProcessGroup(controller.getProcessGroup(),
newParent.getIdentifier())) {
+controller.removeReference(controllerService,
prop.getKey());
+updatedProps.put(prop.getKey().getName(), null);
+}
+}
+}
+}
+if (!updatedProps.isEmpty())
+controllerService.setProperties(updatedProps, true,
Collections.emptySet());
Review Comment:
Fixed, but then removed as it was not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513373146
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##
@@ -2939,6 +2939,115 @@ public ControllerServiceEntity
updateControllerService(final Revision revision,
return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
}
+@Override
+public ControllerServiceEntity moveControllerService(final Revision
revision, final ControllerServiceDTO controllerServiceDTO, final String
newProcessGroupID) {
+// get the component, ensure we have access to it, and perform the
move request
+final ControllerServiceNode controllerService =
controllerServiceDAO.getControllerService(controllerServiceDTO.getId());
+final RevisionUpdate snapshot =
updateComponent(revision,
+controllerService,
+() -> moveControllerServiceWork(controllerService,
newProcessGroupID),
+cs -> {
+awaitValidationCompletion(cs);
Review Comment:
Removed call to wait for validation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513373492
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##
@@ -2939,6 +2939,115 @@ public ControllerServiceEntity
updateControllerService(final Revision revision,
return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
}
+@Override
+public ControllerServiceEntity moveControllerService(final Revision
revision, final ControllerServiceDTO controllerServiceDTO, final String
newProcessGroupID) {
+// get the component, ensure we have access to it, and perform the
move request
+final ControllerServiceNode controllerService =
controllerServiceDAO.getControllerService(controllerServiceDTO.getId());
+final RevisionUpdate snapshot =
updateComponent(revision,
+controllerService,
+() -> moveControllerServiceWork(controllerService,
newProcessGroupID),
+cs -> {
+awaitValidationCompletion(cs);
+final ControllerServiceDTO dto =
dtoFactory.createControllerServiceDto(cs);
+final ControllerServiceReference ref =
controllerService.getReferences();
+final ControllerServiceReferencingComponentsEntity
referencingComponentsEntity =
createControllerServiceReferencingComponentsEntity(ref);
+
dto.setReferencingComponents(referencingComponentsEntity.getControllerServiceReferencingComponents());
+return dto;
+});
+
+final PermissionsDTO permissions =
dtoFactory.createPermissionsDto(controllerService);
+final PermissionsDTO operatePermissions =
dtoFactory.createPermissionsDto(new OperationAuthorizable(controllerService));
+final List bulletins =
dtoFactory.createBulletinDtos(bulletinRepository.findBulletinsForSource(controllerServiceDTO.getId()));
+final List bulletinEntities =
bulletins.stream().map(bulletin -> entityFactory.createBulletinEntity(bulletin,
permissions.getCanRead())).collect(Collectors.toList());
+return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
+}
+
+private ControllerServiceNode moveControllerServiceWork(final
ControllerServiceNode controllerService, final String newProcessGroupID) {
+final ProcessGroup oldParentGroup =
controllerService.getProcessGroup();
+Set referencedComponents =
controllerService.getReferences().getReferencingComponents();
+isReferencesDisabled(referencedComponents);
+oldParentGroup.removeControllerService(controllerService);
+ProcessGroup newParent;
+if (!oldParentGroup.isRootGroup() &&
oldParentGroup.getParent().getIdentifier().equals(newProcessGroupID)) {
+// move to parent process group
+newParent = oldParentGroup.getParent();
+newParent.addControllerService(controllerService);
+
+// unset any references the controller services has to other
controller services that are now out of scope
+Map updatedProps = new HashMap<>();
+Set>
properties = controllerService.getProperties().entrySet();
+for (var prop : properties) {
+var value = prop.getValue();
+if (value !=null) {
+ControllerServiceNode controller;
+try {
+controller =
controllerServiceDAO.getControllerService((value.getRawValue()));
+} catch (Exception e){
+continue;
+}
+if (controller != null) {
+if (!hasProcessGroup(controller.getProcessGroup(),
newParent.getIdentifier())) {
+controller.removeReference(controllerService,
prop.getKey());
+updatedProps.put(prop.getKey().getName(), null);
+}
+}
+}
+}
+if (!updatedProps.isEmpty())
+controllerService.setProperties(updatedProps, true,
Collections.emptySet());
+
+} else {
+// move to child process group
+newParent = oldParentGroup.getProcessGroup(newProcessGroupID);
+newParent.addControllerService(controllerService);
+
+// unset any references for processors that are outside the new
scope
+for (ComponentNode node : referencedComponents) {
+if (!hasProcessGroup(newParent,
node.getProcessGroupIdentifier())) {
+Set>
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513372649
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##
@@ -2939,6 +2939,115 @@ public ControllerServiceEntity
updateControllerService(final Revision revision,
return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
}
+@Override
+public ControllerServiceEntity moveControllerService(final Revision
revision, final ControllerServiceDTO controllerServiceDTO, final String
newProcessGroupID) {
+// get the component, ensure we have access to it, and perform the
move request
+final ControllerServiceNode controllerService =
controllerServiceDAO.getControllerService(controllerServiceDTO.getId());
+final RevisionUpdate snapshot =
updateComponent(revision,
+controllerService,
+() -> moveControllerServiceWork(controllerService,
newProcessGroupID),
+cs -> {
+awaitValidationCompletion(cs);
+final ControllerServiceDTO dto =
dtoFactory.createControllerServiceDto(cs);
+final ControllerServiceReference ref =
controllerService.getReferences();
+final ControllerServiceReferencingComponentsEntity
referencingComponentsEntity =
createControllerServiceReferencingComponentsEntity(ref);
+
dto.setReferencingComponents(referencingComponentsEntity.getControllerServiceReferencingComponents());
+return dto;
+});
+
+final PermissionsDTO permissions =
dtoFactory.createPermissionsDto(controllerService);
+final PermissionsDTO operatePermissions =
dtoFactory.createPermissionsDto(new OperationAuthorizable(controllerService));
+final List bulletins =
dtoFactory.createBulletinDtos(bulletinRepository.findBulletinsForSource(controllerServiceDTO.getId()));
+final List bulletinEntities =
bulletins.stream().map(bulletin -> entityFactory.createBulletinEntity(bulletin,
permissions.getCanRead())).collect(Collectors.toList());
+return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
+}
+
+private ControllerServiceNode moveControllerServiceWork(final
ControllerServiceNode controllerService, final String newProcessGroupID) {
+final ProcessGroup oldParentGroup =
controllerService.getProcessGroup();
+Set referencedComponents =
controllerService.getReferences().getReferencingComponents();
+isReferencesDisabled(referencedComponents);
+oldParentGroup.removeControllerService(controllerService);
+ProcessGroup newParent;
+if (!oldParentGroup.isRootGroup() &&
oldParentGroup.getParent().getIdentifier().equals(newProcessGroupID)) {
+// move to parent process group
+newParent = oldParentGroup.getParent();
+newParent.addControllerService(controllerService);
+
+// unset any references the controller services has to other
controller services that are now out of scope
+Map updatedProps = new HashMap<>();
+Set>
properties = controllerService.getProperties().entrySet();
+for (var prop : properties) {
+var value = prop.getValue();
+if (value !=null) {
+ControllerServiceNode controller;
+try {
+controller =
controllerServiceDAO.getControllerService((value.getRawValue()));
+} catch (Exception e){
+continue;
+}
+if (controller != null) {
+if (!hasProcessGroup(controller.getProcessGroup(),
newParent.getIdentifier())) {
+controller.removeReference(controllerService,
prop.getKey());
+updatedProps.put(prop.getKey().getName(), null);
+}
+}
+}
+}
+if (!updatedProps.isEmpty())
+controllerService.setProperties(updatedProps, true,
Collections.emptySet());
+
+} else {
+// move to child process group
+newParent = oldParentGroup.getProcessGroup(newProcessGroupID);
+newParent.addControllerService(controllerService);
+
+// unset any references for processors that are outside the new
scope
+for (ComponentNode node : referencedComponents) {
+if (!hasProcessGroup(newParent,
node.getProcessGroupIdentifier())) {
+Set>
Re: [PR] NIFI-3785 Added feature to move a controller service to it's parent o… [nifi]
Freedom9339 commented on code in PR #7734:
URL: https://github.com/apache/nifi/pull/7734#discussion_r1513372161
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##
@@ -2939,6 +2939,115 @@ public ControllerServiceEntity
updateControllerService(final Revision revision,
return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
}
+@Override
+public ControllerServiceEntity moveControllerService(final Revision
revision, final ControllerServiceDTO controllerServiceDTO, final String
newProcessGroupID) {
+// get the component, ensure we have access to it, and perform the
move request
+final ControllerServiceNode controllerService =
controllerServiceDAO.getControllerService(controllerServiceDTO.getId());
+final RevisionUpdate snapshot =
updateComponent(revision,
+controllerService,
+() -> moveControllerServiceWork(controllerService,
newProcessGroupID),
+cs -> {
+awaitValidationCompletion(cs);
+final ControllerServiceDTO dto =
dtoFactory.createControllerServiceDto(cs);
+final ControllerServiceReference ref =
controllerService.getReferences();
+final ControllerServiceReferencingComponentsEntity
referencingComponentsEntity =
createControllerServiceReferencingComponentsEntity(ref);
+
dto.setReferencingComponents(referencingComponentsEntity.getControllerServiceReferencingComponents());
+return dto;
+});
+
+final PermissionsDTO permissions =
dtoFactory.createPermissionsDto(controllerService);
+final PermissionsDTO operatePermissions =
dtoFactory.createPermissionsDto(new OperationAuthorizable(controllerService));
+final List bulletins =
dtoFactory.createBulletinDtos(bulletinRepository.findBulletinsForSource(controllerServiceDTO.getId()));
+final List bulletinEntities =
bulletins.stream().map(bulletin -> entityFactory.createBulletinEntity(bulletin,
permissions.getCanRead())).collect(Collectors.toList());
+return
entityFactory.createControllerServiceEntity(snapshot.getComponent(),
dtoFactory.createRevisionDTO(snapshot.getLastModification()), permissions,
operatePermissions, bulletinEntities);
+}
+
+private ControllerServiceNode moveControllerServiceWork(final
ControllerServiceNode controllerService, final String newProcessGroupID) {
Review Comment:
Removed method as it was no longer needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12862 FlowAnalysisResults should not leak anauthorized component details [nifi]
tpalfy commented on PR #8475: URL: https://github.com/apache/nifi/pull/8475#issuecomment-1979418892 > @tpalfy Thanks for the quick turnaround. How would the client know if a user has read access permission to a component? By checking all those field returns on each violation and implicitly determining based on all those fields' values being `null`? I think a better approach would be to include `permissions.canRead` or some other explicit property in the response data model like we do elsewhere. > > cc @mcgilman Check `result.subjectPermission.canRead` on client side and hide/discard the result when false is a good approach. As far as UX goes, that would work without this PR. This PR is needed to make sure that unauthorized data isn't even received by the client at all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12721 Button UX [nifi]
scottyaslan commented on PR #8464: URL: https://github.com/apache/nifi/pull/8464#issuecomment-1979364266 Reviewing... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12720 BUG - Dark mode skeleton loader styles [nifi]
scottyaslan commented on code in PR #8465:
URL: https://github.com/apache/nifi/pull/8465#discussion_r1513275440
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/styles.scss:
##
@@ -334,6 +335,9 @@ $appFontPath: '~roboto-fontface/fonts';
.refresh-timestamp {
color: $warn-palette-A400;
}
+.skeleton-loader{
Review Comment:
Please add a blank line between `.refresh-timestamp` and `.skeleton-loader`
class rules.
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/styles.scss:
##
@@ -334,6 +335,9 @@ $appFontPath: '~roboto-fontface/fonts';
.refresh-timestamp {
color: $warn-palette-A400;
}
+.skeleton-loader{
+background: $canvas-primary-palette-500 !important;
Review Comment:
Two comments here:
1. Please use `$canvas-primary-palette-400`. It is already defined above and
is the color we have been using for similar elements.
2. We typically try to avoid using `!important`. The way to do that is to
have a more specific selector. In this case that will require us to wrap all of
these `@mixin nifi-colors` rules under the `.mat-app-background` class since
that is the class on the ``. So it would look like:
```
.mat-app-background {
a {
color: $accent-palette-A400;
text-decoration-color: $primary-palette-200;
}
a:hover {
text-decoration-color: $accent-palette-A400;
}
.tooltip {
background-color: $canvas-primary-palette-900;
border-color: $canvas-primary-palette-200;
box-shadow: 0 2px 5px $canvas-primary-palette-50;
color: $canvas-primary-palette-200;
}
.property-editor {
background-color: $canvas-primary-palette-900;
box-shadow: 0 2px 5px $canvas-primary-palette-50;
}
.disabled {
color: $primary-palette-500 !important;
fill: $primary-palette-500 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.enabled {
color: $accent-palette-A200 !important;
fill: $accent-palette-A200 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.stopped {
color: $warn-palette-200 !important;
fill: $warn-palette-200 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.running {
color: $canvas-accent-palette-200 !important;
fill: $canvas-accent-palette-200 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.has-errors,
.invalid {
color: $canvas-accent-palette-A200 !important;
fill: $canvas-accent-palette-A200 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.validating {
color: $canvas-primary-palette-400 !important;
fill: $canvas-primary-palette-400 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.transmitting {
color: $accent-palette-A200 !important;
fill: $accent-palette-A200 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.not-transmitting {
color: $primary-palette-500 !important;
fill: $primary-palette-500 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.up-to-date {
color: $canvas-accent-palette-400 !important;
fill: $canvas-accent-palette-400 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.locally-modified,
.sync-failure {
color: $warn-palette-A100 !important;
fill: $warn-palette-A100 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.stale,
.locally-modified-and-stale {
color: $warn-palette-300 !important;
fill: $warn-palette-300 !important;
text-shadow: 0 0 4px $canvas-primary-palette-900;
}
.zero {
color: $accent-palette-A100 !important;
fill: $accent-palette-A100 !important;
text-shadow: none !important;
}
.blank,
.unset,
.sensitive {
color: $canvas-primary-palette-400 !important;
}
button.nifi-button {
border: 1px solid $primary-palette-200;
Re: [PR] NIFI-12862 FlowAnalysisResults should not leak anauthorized component details [nifi]
sardell commented on PR #8475: URL: https://github.com/apache/nifi/pull/8475#issuecomment-1979342161 @tpalfy Thanks for the quick turnaround. How would the client know if a user has read access permission to a component? By checking all those field returns on each violation and implicitly determining based on all those fields values being `null`? I think a better approach would be to include `permissions.canRead` or some other explicit property in the response data model like we do elsewhere. cc @mcgilman -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11520: Add a menu to display Flow Analysis report results [nifi]
tpalfy commented on code in PR #8273:
URL: https://github.com/apache/nifi/pull/8273#discussion_r1513231844
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/controllers/nf-ng-canvas-flow-status-controller.js:
##
@@ -400,6 +406,556 @@
}
}
+/**
+ * The flow analysis controller.
+ */
+
+this.flowAnalysis = {
+
+/**
+ * Create the list of rule violations
+ */
+buildRuleViolationsList: function(rules, violationsAndRecs) {
+var ruleViolationCountEl = $('#rule-violation-count');
+var ruleViolationListEl = $('#rule-violations-list');
+var ruleWarningCountEl = $('#rule-warning-count');
+var ruleWarningListEl = $('#rule-warnings-list');
+var violations = violationsAndRecs.filter(function
(violation) {
+return violation.enforcementPolicy === 'ENFORCE'
+});
+var warnings = violationsAndRecs.filter(function
(violation) {
+return violation.enforcementPolicy === 'WARN'
+});
+ruleViolationCountEl.empty().text('(' + violations.length
+ ')');
+ruleWarningCountEl.empty().text('(' + warnings.length +
')');
+ruleViolationListEl.empty();
+violations.forEach(function(violation) {
+var rule = rules.find(function(rule) {
+return rule.id === violation.ruleId;
+});
+// create DOM elements
+var violationListItemEl = $('');
+var violationEl = $('');
+var violationListItemWrapperEl = $('');
+var violationRuleEl = $('');
+var violationListItemNameEl = $('');
+var violationListItemIdEl = $('');
+var violationInfoButtonEl = $('');
+
+// add text content and button data
+$(violationRuleEl).text(rule.name);
+
$(violationListItemNameEl).text(violation.subjectDisplayName);
Review Comment:
@mcgilman I opened a PR https://issues.apache.org/jira/browse/NIFI-12862
that should take care of this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] NIFI-12862 FlowAnalysisResults should not leak anauthorized component details [nifi]
tpalfy opened a new pull request, #8475: URL: https://github.com/apache/nifi/pull/8475 When building FlowAnalysisRuleViolationDTO objects (in StandardNiFiServiceFacade), violating component details will be left blank when user has no read permission for that component. # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-12862) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2283 Create tool to encrypt sensitive properties in config.yml [nifi-minifi-cpp]
fgerlits commented on code in PR #1725:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1725#discussion_r1513225896
##
libminifi/include/core/ProcessGroup.h:
##
@@ -205,6 +205,8 @@ class ProcessGroup : public CoreComponent {
*/
std::shared_ptr
findControllerService(const std::string ) const;
+ std::vector>
getAllControllerServices() const;
Review Comment:
I have changed this in 929aafecc10ae3d41ce8623d062de58c4f785abe
Now the interface is a bit of a mess, with a mix of return types; I have
created a Jira to fix this: https://issues.apache.org/jira/browse/MINIFICPP-2312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2277 Add virtualenv support for python processors [nifi-minifi-cpp]
lordgamez commented on code in PR #1721:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1721#discussion_r1513224888
##
extensions/python/PythonScriptEngine.cpp:
##
@@ -68,6 +67,73 @@ void initThreads() {
#pragma warning(pop)
#endif
}
+
+std::string encapsulateCommandInQuotesIfNeeded(const std::string& command) {
+#if WIN32
+return "\"" + command + "\"";
+#else
+return command;
+#endif
+}
+
+std::vector getRequirementsFilePaths(const
std::shared_ptr ) {
+ std::vector paths;
+ if (auto python_processor_path =
configuration->get(minifi::Configuration::nifi_python_processor_dir)) {
+for (const auto& entry :
std::filesystem::recursive_directory_iterator(std::filesystem::path{*python_processor_path}))
{
+ if (std::filesystem::is_regular_file(entry.path()) &&
entry.path().filename() == "requirements.txt") {
+paths.push_back(entry.path());
+ }
+}
+ }
+ return paths;
+}
+
+std::string getPythonBinary(const std::shared_ptr ) {
+#if WIN32
+ std::string python_binary = "python";
+#else
+ std::string python_binary = "python3";
+#endif
+ if (auto binary =
configuration->get(minifi::Configuration::nifi_python_env_setup_binary)) {
+python_binary = *binary;
+ }
+ return python_binary;
+}
+
+void createVirtualEnvIfSpecified(const std::shared_ptr
) {
+ if (auto path =
configuration->get(minifi::Configuration::nifi_python_virtualenv_directory)) {
+PythonConfigState::getInstance().virtualenv_path = *path;
+if
(!std::filesystem::exists(PythonConfigState::getInstance().virtualenv_path) ||
!std::filesystem::is_empty(PythonConfigState::getInstance().virtualenv_path)) {
+ auto venv_command = "\"" +
PythonConfigState::getInstance().python_binary + "\" -m venv \"" +
PythonConfigState::getInstance().virtualenv_path.string() + "\"";
+ auto return_value =
std::system(encapsulateCommandInQuotesIfNeeded(venv_command).c_str());
+ if (return_value != 0) {
+throw PythonScriptException(fmt::format("The following command
creating python virtual env failed: '{}'", venv_command));
+ }
+}
+ }
+}
+
+void installPythonPackagesIfRequested(const std::shared_ptr
, const std::shared_ptr& logger) {
+ std::string automatic_install_str;
+ if (!PythonConfigState::getInstance().isPackageInstallationNeeded()) {
+return;
+ }
+ auto requirement_file_paths = getRequirementsFilePaths(configuration);
+ for (const auto& requirements_file_path : requirement_file_paths) {
+logger->log_info("Installing python packages from the following
requirements.txt file: {}", requirements_file_path.string());
+std::string pip_command;
+#if WIN32
+
pip_command.append("\"").append((PythonConfigState::getInstance().virtualenv_path
/ "Scripts" / "activate.bat").string()).append("\" && ");
+#else
+pip_command.append(".
\"").append((PythonConfigState::getInstance().virtualenv_path / "bin" /
"activate").string()).append("\" && ");
+#endif
+
pip_command.append("\"").append(PythonConfigState::getInstance().python_binary).append("\"
-m pip install --no-cache-dir -r
\"").append(requirements_file_path.string()).append("\"");
+auto return_value =
std::system(encapsulateCommandInQuotesIfNeeded(pip_command).c_str());
Review Comment:
Yes it looks strange, but there is an issue with std::system on Windows that
needs the additional quotes on the whole command to work correctly. If I
remember well it passes the system command as a parameter to the `cmd.exe /C`
command and the additional quotes are needed to parse the system command's
arguments correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2277 Add virtualenv support for python processors [nifi-minifi-cpp]
lordgamez commented on code in PR #1721:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1721#discussion_r1513218719
##
docker/python-verify/conda.Dockerfile:
##
@@ -31,14 +31,12 @@ USER root
RUN wget https://repo.anaconda.com/archive/Anaconda3-2023.09-0-Linux-x86_64.sh
-P /tmp \
&& echo "6c8a4abb36fbb711dc055b7049a23bbfd61d356de9468b41c5140f8a11abd851
/tmp/Anaconda3-2023.09-0-Linux-x86_64.sh" | sha256sum -c \
&& bash /tmp/Anaconda3-2023.09-0-Linux-x86_64.sh -b -p /opt/conda \
-&& chown -R ${USER}:${USER} /opt/conda \
-&& mkdir /home/${USER} \
-&& chown -R ${USER}:${USER} /home/${USER}
+&& chown -R ${USER}:${USER} /opt/conda
USER ${USER}
RUN ${CONDA_HOME}/bin/conda init bash
-RUN ${CONDA_HOME}/bin/conda install langchain -c conda-forge
+RUN ${CONDA_HOME}/bin/conda install "langchain<=0.17.0" -c conda-forge
Review Comment:
Currently the latest version of langchain is 0.17.0 and as we run the python
tests on multiple distributions and with virtualenvs or with conda in the CI,
we run these tests with version 0.17.0 or bwloe. The tests with these versions
are tested and passing, but the langchain library is rapidly changing so we
don't want to use the latest version in the tests because they may break with a
future version, so it is better to have the version fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (MINIFICPP-2312) Do not return shared_ptrs unnecessarily in ControllerServiceMap and ControllerServiceNode
[
https://issues.apache.org/jira/browse/MINIFICPP-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Gerlits updated MINIFICPP-2312:
--
Description:
There are several functions in and around Controller Service storage which
return shared_ptr's, probably unnecessarily. For example,
{noformat}
ControllerServiceMap::getAllControllerServices()
ControllerServiceNode::getControllerServiceImplementation();
ControllerServiceNode::getLinkedControllerServices();
ControllerServiceNode::getLinkedComponents();
{noformat}
Check if these function can return non-owning, possibly const, possibly
gsl::not_null, pointers instead of shared_ptrs, and if it is possible, then
change them.
was:
There are several functions in and around Controller Service storage which
return shared_ptr's, probably unnecessarily. For example,
{noformat}
ControllerServiceMap::getAllControllerServices()
ControllerServiceNode::getControllerServiceImplementation();
ControllerServiceNode::getLinkedControllerServices();
ControllerServiceNode::getLinkedComponents();
{noformat}
Check if these function can return non-owning, possibly const, possibly
gsl::non_null, pointers instead of shared_ptrs, and if it is possible, then
change them.
> Do not return shared_ptrs unnecessarily in ControllerServiceMap and
> ControllerServiceNode
> -
>
> Key: MINIFICPP-2312
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2312
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Ferenc Gerlits
>Assignee: Ferenc Gerlits
>Priority: Minor
>
> There are several functions in and around Controller Service storage which
> return shared_ptr's, probably unnecessarily. For example,
> {noformat}
> ControllerServiceMap::getAllControllerServices()
> ControllerServiceNode::getControllerServiceImplementation();
> ControllerServiceNode::getLinkedControllerServices();
> ControllerServiceNode::getLinkedComponents();
> {noformat}
> Check if these function can return non-owning, possibly const, possibly
> gsl::not_null, pointers instead of shared_ptrs, and if it is possible, then
> change them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2312) Do not return shared_ptrs unnecessarily in ControllerServiceMap and ControllerServiceNode
[
https://issues.apache.org/jira/browse/MINIFICPP-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Gerlits updated MINIFICPP-2312:
--
Description:
There are several functions in and around Controller Service storage which
return shared_ptr's, probably unnecessarily. For example,
{noformat}
ControllerServiceMap::getAllControllerServices()
ControllerServiceNode::getControllerServiceImplementation();
ControllerServiceNode::getLinkedControllerServices();
ControllerServiceNode::getLinkedComponents();
{noformat}
Check if these function can return non-owning, possibly const, possibly
gsl::non_null, pointers instead of shared_ptrs, and if it is possible, then
change them.
was:
There are several functions in and around Controller Service strorage which
return shared_ptr's, probably unnecessarily. For example,
{noformat}
ControllerServiceMap::getAllControllerServices()
ControllerServiceNode::getControllerServiceImplementation();
ControllerServiceNode::getLinkedControllerServices();
ControllerServiceNode::getLinkedComponents();
{noformat}
Check if these function can return non-owning, possibly const, possibly
gsl::non_null, pointers instead of shared_ptrs, and if it is possible, then
change them.
> Do not return shared_ptrs unnecessarily in ControllerServiceMap and
> ControllerServiceNode
> -
>
> Key: MINIFICPP-2312
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2312
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Ferenc Gerlits
>Assignee: Ferenc Gerlits
>Priority: Minor
>
> There are several functions in and around Controller Service storage which
> return shared_ptr's, probably unnecessarily. For example,
> {noformat}
> ControllerServiceMap::getAllControllerServices()
> ControllerServiceNode::getControllerServiceImplementation();
> ControllerServiceNode::getLinkedControllerServices();
> ControllerServiceNode::getLinkedComponents();
> {noformat}
> Check if these function can return non-owning, possibly const, possibly
> gsl::non_null, pointers instead of shared_ptrs, and if it is possible, then
> change them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (MINIFICPP-2312) Do not return shared_ptrs unnecessarily in ControllerServiceMap and ControllerServiceNode
Ferenc Gerlits created MINIFICPP-2312:
-
Summary: Do not return shared_ptrs unnecessarily in
ControllerServiceMap and ControllerServiceNode
Key: MINIFICPP-2312
URL: https://issues.apache.org/jira/browse/MINIFICPP-2312
Project: Apache NiFi MiNiFi C++
Issue Type: Improvement
Reporter: Ferenc Gerlits
Assignee: Ferenc Gerlits
There are several functions in and around Controller Service strorage which
return shared_ptr's, probably unnecessarily. For example,
{noformat}
ControllerServiceMap::getAllControllerServices()
ControllerServiceNode::getControllerServiceImplementation();
ControllerServiceNode::getLinkedControllerServices();
ControllerServiceNode::getLinkedComponents();
{noformat}
Check if these function can return non-owning, possibly const, possibly
gsl::non_null, pointers instead of shared_ptrs, and if it is possible, then
change them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] MINIFICPP-2283 Create tool to encrypt sensitive properties in config.yml [nifi-minifi-cpp]
szaszm commented on code in PR #1725:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1725#discussion_r1513209019
##
libminifi/src/core/json/JsonFlowSerializer.cpp:
##
@@ -47,14 +47,21 @@ void
JsonFlowSerializer::encryptSensitiveProperties(rapidjson::Value _j
}
if (properties.at(name).isSensitive()) {
auto& value = property.value;
- const std::string_view value_sv{value.GetString(),
value.GetStringLength()};
+ const std::string_view value_sv = component_overrides.contains(name) ?
component_overrides.at(name) : std::string_view{value.GetString(),
value.GetStringLength()};
const std::string encrypted_value =
utils::crypto::property_encryption::encrypt(value_sv, encryption_provider);
value.SetString(encrypted_value.c_str(), encrypted_value.size(), alloc);
}
+component_overrides.erase(name);
+ }
+
+ for (const auto& [name, value] : component_overrides) {
+const std::string encrypted_value =
utils::crypto::property_encryption::encrypt(value, encryption_provider);
+property_jsons.AddMember(rapidjson::Value(name, alloc),
rapidjson::Value(encrypted_value, alloc), alloc);
Review Comment:
My bad, I missed the encrypt call on line 51 at first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2277 Add virtualenv support for python processors [nifi-minifi-cpp]
fgerlits commented on code in PR #1721:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1721#discussion_r1512842925
##
extensions/python/PYTHON.md:
##
@@ -155,10 +158,33 @@ In the flow configuration these Python processors can be
referenced by their ful
Due to some differences between the NiFi and MiNiFi C++ processors and
implementation, there are some limitations using the NiFi Python processors:
- Record based processors are not yet supported in MiNiFi C++, so the NiFi
Python processors inherited from RecordTransform are not supported.
-- Virtualenv support is not yet available in MiNiFi C++, so all required
packages must be installed on the system.
- Controller properties are not supported at the moment.
- There are some validators in NiFi that are not present in MiNiFi C++, so
some property validations will be missing using the NiFi Python processors.
- Allowable values specified in NiFi Python processors are ignored in MiNiFi
C++ (due to MiNiFi C++ requiring them to be specified at compile time), so the
property values are not pre-verified.
- MiNiFi C++ only supports expression language with flow file attributes, so
only FLOWFILE_ATTRIBUTES expression language scope is supported, otherwise the
expression language will not be evaluated.
- MiNiFi C++ does not support property dependencies, so the property
dependencies will be ignored. If a property depends on another property, the
property will not be required.
- MiNiFi C++ does not support the use of self.jvm member in Python processors
that provides JVM bindings in NiFi, it is set to None in MiNiFi C++.
+- Inline definition of Python package dependencies, defined in the
ProcessorDetails nested class are not supported as in NiFi, so the dependencies
must be defined in the requirements.txt files. If a processor's dependencies
are defined in the ProcessorDetails class, the dependencies should be copied to
the requirements.txt file.
+
+## Use Python processors from virtualenv
+
+It is possible to set a virtualenv to be used by the Python processors in
Apache MiNiFi C++. If the virtualenv directory is set, the Python processors
will be executed using the packages installed in the virtualenv. If the
virtualenv directory is not set, the Python processors will be executed using
the packages installed on the system.
+
+# in minifi.properties
+nifi.python.virtualenv.directory=${MINIFI_HOME}/minifi-python-env
+
+**NOTE:* Using different python versions for the system and the virtualenv is
not supported. The virtualenv must be created using the same python version as
the system python.
+
+## Automatically install dependencies from requirements.txt files
+
+It is possible to automatically install the dependencies of the Python
processors in the virtualenv defined in requirements.txt files. To enable this
feature, the `nifi.python.install.packages.automatically` property must be set
to true. If this property is set to true, then all requirements.txt files that
appear under the MiNiFi Python directory and its subdirectories (defined by the
`nifi.python.processor.dir` property) will be used to install the Python
packages. If the `nifi.python.virtualenv.directory` property is set, the
packages are installed in the virtualenv, otherwise this option is ignored. Due
to install schema differences in different platforms, system level packages are
expected to be installed manually by the user.
Review Comment:
I find "... otherwise this option is ignored" unclear (which option is
"this"?) If it means that for automatic installation, both properties need to
be set, I would write:
```suggestion
It is possible to automatically install the dependencies of the Python
processors defined in requirements.txt files into a virtualenv. To enable this
feature, the `nifi.python.install.packages.automatically` property must be set
to true, and the `nifi.python.virtualenv.directory` property must be set to a
directory where a virtualenv either already exists, or it can be set up. In
this case, all requirements.txt files that appear under the MiNiFi Python
directory (defined by the `nifi.python.processor.dir` property) and its
subdirectories will be used to install the Python packages into the given
virtualenv. Due to install schema differences in different platforms, system
level packages are expected to be installed manually by the user.
```
##
extensions/python/PythonScriptEngine.cpp:
##
@@ -68,6 +67,73 @@ void initThreads() {
#pragma warning(pop)
#endif
}
+
+std::string encapsulateCommandInQuotesIfNeeded(const std::string& command) {
+#if WIN32
+return "\"" + command + "\"";
+#else
+return command;
+#endif
+}
+
+std::vector getRequirementsFilePaths(const
std::shared_ptr ) {
+ std::vector paths;
+ if (auto python_processor_path =
configuration->get(minifi::Configuration::nifi_python_processor_dir)) {
+for (const auto& entry :
Re: [PR] MINIFICPP-2203 Add support for building Windows MSI without any redistributables included [nifi-minifi-cpp]
lordgamez commented on code in PR #1734:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1734#discussion_r1509271721
##
cmake/MiNiFiOptions.cmake:
##
@@ -82,6 +80,8 @@ list(APPEND STRICT_GSL_CHECKS_Values AUDIT ON DEBUG_ONLY OFF)
set_property(CACHE STRICT_GSL_CHECKS PROPERTY STRINGS
${STRICT_GSL_CHECKS_Values})
if (WIN32)
+add_minifi_option(INSTALLER_MERGE_MODULES "Creates installer with merge
modules" OFF)
+add_minifi_option(INSTALLER_WITH_VC_REDISTRIBUTABLES "Creates installer
with Visual C++ redistributables included" OFF)
Review Comment:
I agree the naming should be changed, to be more clear, and clarify which
option bundles what artifacts exactly. It also needs to be specified if a
bundle falls under a different license.
On the other hand I would keep all three bundle options separate. The
rationale behind this version is it's easier and more direct to specify a
single option for each installer bundle option instead of combining build
options like `INCLUDE_MS_BLOBS` + `USE_UCRT_DLLS`. So I would keep them
separately like `INCLUDE_UCRT_DLLS`, `INCLUDE_VC_REDIST_DLLS`, and
`INCLUDE_VC_REDIST_MERGE_MODULES`. This way you can directly specify which
libraries you want to include in the installer, and by not specifying anything,
no Microsoft blobs will be included. What are your thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-12630) NPE getLogger in ConsumeSlack
[
https://issues.apache.org/jira/browse/NIFI-12630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823702#comment-17823702
]
Jim Steinebrey commented on NIFI-12630:
---
FYI: AbstractSessionFactoryProcessor.getLogger() can not be called during the
instantiation of a processor because getLogger() returns null until
AbstractSessionFactoryProcessor.initialize(final
ProcessorInitializationContext context) is executed.
This bug occurred because ConsumeSlack (and PublishSlack) had a instance
variable initialized like this:
public class ConsumeSlack extends AbstractProcessor implements
VerifiableProcessor {
...
private final RateLimit rateLimit = new RateLimit(getLogger());
...
}
> NPE getLogger in ConsumeSlack
> -
>
> Key: NIFI-12630
> URL: https://issues.apache.org/jira/browse/NIFI-12630
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 2.0.0-M1, 1.24.0
>Reporter: Pierre Villard
>Assignee: Jim Steinebrey
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Getting this NPE when hitting the Slack rate limit:
> {code:java}
> 2024-01-18 16:07:53,560 WARN [Timer-Driven Process Thread-12]
> o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception
> in Component [ConsumeSlack[id=592d68c7-3fe6-3039-53e4-eae3bfbfbd57]]
> java.lang.NullPointerException: Cannot invoke
> "org.apache.nifi.logging.ComponentLog.debug(String, Object[])" because
> "this.logger" is null
> at
> org.apache.nifi.processors.slack.util.RateLimit.isLimitReached(RateLimit.java:42)
> at
> org.apache.nifi.processors.slack.ConsumeSlack.onTrigger(ConsumeSlack.java:332)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (NIFI-12630) NPE getLogger in ConsumeSlack
[
https://issues.apache.org/jira/browse/NIFI-12630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Steinebrey reassigned NIFI-12630:
-
Assignee: Jim Steinebrey
> NPE getLogger in ConsumeSlack
> -
>
> Key: NIFI-12630
> URL: https://issues.apache.org/jira/browse/NIFI-12630
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 2.0.0-M1, 1.24.0
>Reporter: Pierre Villard
>Assignee: Jim Steinebrey
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Getting this NPE when hitting the Slack rate limit:
> {code:java}
> 2024-01-18 16:07:53,560 WARN [Timer-Driven Process Thread-12]
> o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception
> in Component [ConsumeSlack[id=592d68c7-3fe6-3039-53e4-eae3bfbfbd57]]
> java.lang.NullPointerException: Cannot invoke
> "org.apache.nifi.logging.ComponentLog.debug(String, Object[])" because
> "this.logger" is null
> at
> org.apache.nifi.processors.slack.util.RateLimit.isLimitReached(RateLimit.java:42)
> at
> org.apache.nifi.processors.slack.ConsumeSlack.onTrigger(ConsumeSlack.java:332)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12630 Fix NPE getLogger in ConsumeSlack and PublishSlack [nifi]
jrsteinebrey opened a new pull request, #8474: URL: https://github.com/apache/nifi/pull/8474 # Summary [NIFI-12630](https://issues.apache.org/jira/browse/NIFI-12630) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-12630) issue created ### Pull Request Tracking - [X ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification I added a new unit test to confirm the change works ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing No new files and no new dependencies ### Documentation No documentation changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2203 Add support for building Windows MSI without any redistributables included [nifi-minifi-cpp]
martinzink commented on code in PR #1734:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1734#discussion_r1513120771
##
cmake/MiNiFiOptions.cmake:
##
@@ -82,6 +80,8 @@ list(APPEND STRICT_GSL_CHECKS_Values AUDIT ON DEBUG_ONLY OFF)
set_property(CACHE STRICT_GSL_CHECKS PROPERTY STRINGS
${STRICT_GSL_CHECKS_Values})
if (WIN32)
+add_minifi_option(INSTALLER_MERGE_MODULES "Creates installer with merge
modules" OFF)
+add_minifi_option(INSTALLER_WITH_VC_REDISTRIBUTABLES "Creates installer
with Visual C++ redistributables included" OFF)
Review Comment:
Could we add the MINIFI_ prefix to these new options?
(In my opinion we should add that to every option thats ours so it doesnt
conflict with thirdparty options and it makes it easier to filter out
thirdparty related options (e.g. python bootstrap).)
So adding it to new flags should be good way to start this transition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-1797 Python bootstrap (part 1) [nifi-minifi-cpp]
martinzink commented on PR #1681: URL: https://github.com/apache/nifi-minifi-cpp/pull/1681#issuecomment-1979056150 > In the same Windows environment I had issues with `vswhere`, I also had issue with the compiler command line after installing vswhere with scoop and refusing the offered choco install. > > ``` > -- The C compiler identification is unknown > -- The CXX compiler identification is unknown > CMake Error at CMakeLists.txt:25 (project): > The CMAKE_C_COMPILER: > > C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.38.33130/bin/Hostx64/x64/cl.exe > > is not a full path to an existing compiler tool. > > Tell CMake where to find the compiler by setting either the environment > variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to > the compiler, or to the compiler name if it is in the PATH. > > > CMake Error at CMakeLists.txt:25 (project): > The CMAKE_CXX_COMPILER: > > C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.38.33130/bin/Hostx64/x64/cl.exe > > is not a full path to an existing compiler tool. > > Tell CMake where to find the compiler by setting either the environment > variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path > to the compiler, or to the compiler name if it is in the PATH. > > > -- Configuring incomplete, errors occurred! > ``` This looks interesting 樂 , does the qouted path look correct? Is it the same that vswhere returns? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12720 BUG - Dark mode skeleton loader styles [nifi]
scottyaslan commented on PR #8465: URL: https://github.com/apache/nifi/pull/8465#issuecomment-1979055485 Reviewing... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-1797 Python bootstrap (part 1) [nifi-minifi-cpp]
martinzink commented on code in PR #1681:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1681#discussion_r1513040448
##
bootstrap/package_manager.py:
##
@@ -0,0 +1,259 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import os
+import platform
+import subprocess
+import sys
+from typing import Dict, Set
+
+from distro import distro
+
+
+def _query_yes_no(question: str, no_confirm: bool) -> bool:
+valid = {"yes": True, "y": True, "ye": True, "no": False, "n": False}
+
+if no_confirm:
+print("Running {} with noconfirm".format(question))
+return True
+while True:
+print("{} [y/n]".format(question), end=' ', flush=True)
+choice = input().lower()
+if choice in valid:
+return valid[choice]
+else:
+print("Please respond with 'yes' or 'no' " "(or 'y' or 'n').")
+
+
+def _run_command_with_confirm(command: str, no_confirm: bool) -> bool:
+if _query_yes_no("Running {}".format(command), no_confirm):
+return os.system(command) == 0
+
+
+class PackageManager(object):
+def __init__(self, no_confirm):
+self.no_confirm = no_confirm
+pass
+
+def install(self, dependencies: Dict[str, Set[str]]) -> bool:
+raise Exception("NotImplementedException")
+
+def install_compiler(self) -> str:
+raise Exception("NotImplementedException")
+
+def _install(self, dependencies: Dict[str, Set[str]], replace_dict:
Dict[str, Set[str]], install_cmd: str) -> bool:
+dependencies.update({k: v for k, v in replace_dict.items() if k in
dependencies})
+dependencies = self._filter_out_installed_packages(dependencies)
+dependencies_str = " ".join(str(value) for value_set in
dependencies.values() for value in value_set)
+if not dependencies_str or dependencies_str.isspace():
+return True
+return _run_command_with_confirm(f"{install_cmd} {dependencies_str}",
self.no_confirm)
+
+def _get_installed_packages(self) -> Set[str]:
+raise Exception("NotImplementedException")
+
+def _filter_out_installed_packages(self, dependencies: Dict[str,
Set[str]]):
+installed_packages = self._get_installed_packages()
+filtered_packages = {k: (v - installed_packages) for k, v in
dependencies.items()}
+for installed_package in installed_packages:
+filtered_packages.pop(installed_package, None)
+return filtered_packages
+
+def run_cmd(self, cmd: str) -> bool:
+result = subprocess.run(f"{cmd}", shell=True, text=True)
+return result.returncode == 0
+
+
+class BrewPackageManager(PackageManager):
+def __init__(self, no_confirm):
+PackageManager.__init__(self, no_confirm)
+
+def install(self, dependencies: Dict[str, Set[str]]) -> bool:
+return self._install(dependencies=dependencies,
+ install_cmd="brew install",
+ replace_dict={"patch": set(),
+ "jni": {"maven"}})
+
+def install_compiler(self) -> str:
+self.install({"compiler": set()})
+return ""
+
+def _get_installed_packages(self) -> Set[str]:
+result = subprocess.run(['brew', 'list'], text=True,
capture_output=True, check=True)
+lines = result.stdout.splitlines()
+lines = [line.split('@', 1)[0] for line in lines]
+return set(lines)
+
+
+class AptPackageManager(PackageManager):
+def __init__(self, no_confirm):
+PackageManager.__init__(self, no_confirm)
+
+def install(self, dependencies: Dict[str, Set[str]]) -> bool:
+return self._install(dependencies=dependencies,
+ install_cmd="sudo apt install -y",
+ replace_dict={"libarchive": {"liblzma-dev"},
+ "lua": {"liblua5.1-0-dev"},
+ "python": {"libpython3-dev"},
+ "libusb": {"libusb-1.0-0-dev",
"libusb-dev"},
+ "libpng": {"libpng-dev"},
+ "libpcap": {"libpcap-dev"},
+
[jira] [Assigned] (NIFI-12860) Null exception from ExtensionMetadata caused by NIFI-12113
[ https://issues.apache.org/jira/browse/NIFI-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-12860: --- Assignee: Peter Radden > Null exception from ExtensionMetadata caused by NIFI-12113 > -- > > Key: NIFI-12860 > URL: https://issues.apache.org/jira/browse/NIFI-12860 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.24.0, 1.25.0, 2.0.0-M2 >Reporter: Peter Radden >Assignee: Peter Radden >Priority: Minor > Fix For: 2.0.0, 1.26.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Shown using command similar to: > cli.sh registry list-extensions -ot json -p tk.properties > ERROR: Error executing command 'list-extensions' : null > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12860) Null exception from ExtensionMetadata caused by NIFI-12113
[ https://issues.apache.org/jira/browse/NIFI-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-12860. - Fix Version/s: 2.0.0 1.26.0 Resolution: Fixed > Null exception from ExtensionMetadata caused by NIFI-12113 > -- > > Key: NIFI-12860 > URL: https://issues.apache.org/jira/browse/NIFI-12860 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.24.0, 1.25.0, 2.0.0-M2 >Reporter: Peter Radden >Priority: Minor > Fix For: 2.0.0, 1.26.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Shown using command similar to: > cli.sh registry list-extensions -ot json -p tk.properties > ERROR: Error executing command 'list-extensions' : null > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12860) Null exception from ExtensionMetadata caused by NIFI-12113
[ https://issues.apache.org/jira/browse/NIFI-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823677#comment-17823677 ] ASF subversion and git services commented on NIFI-12860: Commit 785039e1d0d9e33875622be0d7cfb2d4d42a2b37 in nifi's branch refs/heads/support/nifi-1.x from Peter Radden [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=785039e1d0 ] NIFI-12860 Fixed NPE in ExtensionMetadata Constructor This closes #8467 Signed-off-by: David Handermann > Null exception from ExtensionMetadata caused by NIFI-12113 > -- > > Key: NIFI-12860 > URL: https://issues.apache.org/jira/browse/NIFI-12860 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.24.0, 1.25.0, 2.0.0-M2 >Reporter: Peter Radden >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > Shown using command similar to: > cli.sh registry list-extensions -ot json -p tk.properties > ERROR: Error executing command 'list-extensions' : null > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12860 Null exception from ExtensionMetadata caused by NIFI-12113 [nifi]
exceptionfactory commented on PR #8467: URL: https://github.com/apache/nifi/pull/8467#issuecomment-1979032317 Thanks @pradden, merged in commit https://github.com/apache/nifi/commit/785039e1d0d9e33875622be0d7cfb2d4d42a2b37 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12860 Null exception from ExtensionMetadata caused by NIFI-12113 [nifi]
exceptionfactory closed pull request #8467: NIFI-12860 Null exception from ExtensionMetadata caused by NIFI-12113 URL: https://github.com/apache/nifi/pull/8467 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12860) Null exception from ExtensionMetadata caused by NIFI-12113
[ https://issues.apache.org/jira/browse/NIFI-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823673#comment-17823673 ] ASF subversion and git services commented on NIFI-12860: Commit 264ea30bbd3144b15a87fcb3195a0f3a41b23c8f in nifi's branch refs/heads/main from Peter Radden [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=264ea30bbd ] NIFI-12860 Fixed NPE in ExtensionMetadata Constructor This closes #8468 Signed-off-by: David Handermann > Null exception from ExtensionMetadata caused by NIFI-12113 > -- > > Key: NIFI-12860 > URL: https://issues.apache.org/jira/browse/NIFI-12860 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.24.0, 1.25.0, 2.0.0-M2 >Reporter: Peter Radden >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > Shown using command similar to: > cli.sh registry list-extensions -ot json -p tk.properties > ERROR: Error executing command 'list-extensions' : null > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12860 Null exception from ExtensionMetadata caused by NIFI-12113 [nifi]
exceptionfactory closed pull request #8468: NIFI-12860 Null exception from ExtensionMetadata caused by NIFI-12113 URL: https://github.com/apache/nifi/pull/8468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12846) AWS Assume Role Credentials with VPCE Endpoint URL cannot handle the Region
[
https://issues.apache.org/jira/browse/NIFI-12846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-12846:
--
Fix Version/s: 1.26.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> AWS Assume Role Credentials with VPCE Endpoint URL cannot handle the Region
>
>
> Key: NIFI-12846
> URL: https://issues.apache.org/jira/browse/NIFI-12846
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.25.0
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
> Fix For: 1.26.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case of custom Endpoint URLs, the AWS client library may be able to parse
> the Region from the URL but can not handle VPCE URLs (e.g.
> [https://vpce-{*}-{*}***-eu-central-1a.sts.eu-central-1.vpce.amazonaws.com).|https://vpce-%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A-%2A%2A%2A%2A%2A%2A%2A%2A-eu-central-1a.sts.eu-central-1.vpce.amazonaws.com)./]
> {code:java}
> 2024-02-27 13:13:04,102 ERROR [Timer-Driven Process Thread-1]
> o.apache.nifi.processors.aws.s3.ListS3
> ListS3[id=d5e08c19-a155-3b34-e9e6-dbd70e048cd1] Failed to list contents of
> bucket
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> Credential should be scoped to a valid region. (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code: SignatureDoesNotMatch;
> Request ID: 7820b219-dee5-4989-8d0c-46523
> 1469705; Proxy: null)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1731)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1698)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1687)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:532)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:501)
> ... {code}
> Use the explicit Region property (added in NIFI-10791) for VPCE endpoints.
> The issue affects the 1.x line only. In NiFi 2.0, the AWS processors
> (including the credential handling) have been refactored (NIFI-12144).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12846) AWS Assume Role Credentials with VPCE Endpoint URL cannot handle the Region
[
https://issues.apache.org/jira/browse/NIFI-12846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823651#comment-17823651
]
ASF subversion and git services commented on NIFI-12846:
Commit 40f2144942e5ab75c1464f48873cfa415ab1aecb in nifi's branch
refs/heads/support/nifi-1.x from Peter Turcsanyi
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=40f2144942 ]
NIFI-12846 Fixed Region handling for AWS Assume Role Credentials with VPCE
Endpoint URL
Signed-off-by: Pierre Villard
This closes #8473.
> AWS Assume Role Credentials with VPCE Endpoint URL cannot handle the Region
>
>
> Key: NIFI-12846
> URL: https://issues.apache.org/jira/browse/NIFI-12846
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.25.0
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case of custom Endpoint URLs, the AWS client library may be able to parse
> the Region from the URL but can not handle VPCE URLs (e.g.
> [https://vpce-{*}-{*}***-eu-central-1a.sts.eu-central-1.vpce.amazonaws.com).|https://vpce-%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A-%2A%2A%2A%2A%2A%2A%2A%2A-eu-central-1a.sts.eu-central-1.vpce.amazonaws.com)./]
> {code:java}
> 2024-02-27 13:13:04,102 ERROR [Timer-Driven Process Thread-1]
> o.apache.nifi.processors.aws.s3.ListS3
> ListS3[id=d5e08c19-a155-3b34-e9e6-dbd70e048cd1] Failed to list contents of
> bucket
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> Credential should be scoped to a valid region. (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code: SignatureDoesNotMatch;
> Request ID: 7820b219-dee5-4989-8d0c-46523
> 1469705; Proxy: null)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1731)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1698)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1687)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:532)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:501)
> ... {code}
> Use the explicit Region property (added in NIFI-10791) for VPCE endpoints.
> The issue affects the 1.x line only. In NiFi 2.0, the AWS processors
> (including the credential handling) have been refactored (NIFI-12144).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12846 Fixed Region handling for AWS Assume Role Credentials with… [nifi]
pvillard31 commented on PR #8473: URL: https://github.com/apache/nifi/pull/8473#issuecomment-1978920487 Merged in 1.x via 436f63c52b62a37c256a272b107cce380414df25 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12846 Fixed Region handling for AWS Assume Role Credentials with… [nifi]
pvillard31 closed pull request #8473: NIFI-12846 Fixed Region handling for AWS Assume Role Credentials with… URL: https://github.com/apache/nifi/pull/8473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]
exceptionfactory commented on PR #8407: URL: https://github.com/apache/nifi/pull/8407#issuecomment-1978901124 Thanks for the feedback @markap14, I rebased to incorporate the recent Python framework improvements, and also addressed the comments regarding log writing and exception handling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]
exceptionfactory commented on code in PR #8407:
URL: https://github.com/apache/nifi/pull/8407#discussion_r1512922173
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcessLogReader.java:
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.py4j;
+
+import org.apache.nifi.py4j.logging.PythonLogLevel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.stream.Collectors;
+
+/**
+ * Runnable Command for reading a line from Process Output Stream and writing
to a Logger
+ */
+class PythonProcessLogReader implements Runnable {
+private static final int LOG_LEVEL_BEGIN_INDEX = 0;
+
+private static final int LOG_LEVEL_END_INDEX = 2;
+
+private static final int MESSAGE_BEGIN_INDEX = 3;
+
+private static final char NAME_MESSAGE_SEPARATOR = ':';
+
+private static final int MINIMUM_LOGGER_NAME_INDEX = 3;
+
+private static final String LOG_PREFIX = "PY4JLOG";
+
+private static final int PREFIXED_LOG_LEVEL_BEGIN_INDEX = 8;
+
+private static final String LINE_SEPARATOR = System.lineSeparator();
+
+private static final Map PYTHON_LOG_LEVELS =
Arrays.stream(PythonLogLevel.values()).collect(
+Collectors.toUnmodifiableMap(
+pythonLogLevel ->
Integer.toString(pythonLogLevel.getLevel()),
+pythonLogLevel -> pythonLogLevel
+)
+);
+
+private final Logger processLogger =
LoggerFactory.getLogger("org.apache.nifi.py4j.ProcessLog");
+
+private final BufferedReader processReader;
+
+/**
+ * Standard constructor with Buffered Reader connected to Python Process
Output Stream
+ *
+ * @param processReader Reader from Process Output Stream
+ */
+PythonProcessLogReader(final BufferedReader processReader) {
+this.processReader = Objects.requireNonNull(processReader, "Reader
required");
+}
+
+/**
+ * Read lines from Process Reader and write log messages based on parsed
level and named logger
+ */
+@Override
+public void run() {
+final Queue parsedRecords = new LinkedList<>();
+
+try {
+String line = processReader.readLine();
+while (line != null) {
+processLine(line, parsedRecords);
+
+if (parsedRecords.size() == 2 || !processReader.ready()) {
+final ParsedRecord parsedRecord = parsedRecords.remove();
+log(parsedRecord);
+}
Review Comment:
Thanks for the thorough feedback and testing. I made a change to check the
readiness of the Process Reader separately, and then the example `LogContents`
Processor writes the messages as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2283 Create tool to encrypt sensitive properties in config.yml [nifi-minifi-cpp]
szaszm commented on code in PR #1725:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1725#discussion_r1512874899
##
libminifi/src/core/json/JsonFlowSerializer.cpp:
##
@@ -47,14 +47,21 @@ void
JsonFlowSerializer::encryptSensitiveProperties(rapidjson::Value _j
}
if (properties.at(name).isSensitive()) {
auto& value = property.value;
- const std::string_view value_sv{value.GetString(),
value.GetStringLength()};
+ const std::string_view value_sv = component_overrides.contains(name) ?
component_overrides.at(name) : std::string_view{value.GetString(),
value.GetStringLength()};
const std::string encrypted_value =
utils::crypto::property_encryption::encrypt(value_sv, encryption_provider);
value.SetString(encrypted_value.c_str(), encrypted_value.size(), alloc);
}
+component_overrides.erase(name);
+ }
+
+ for (const auto& [name, value] : component_overrides) {
+const std::string encrypted_value =
utils::crypto::property_encryption::encrypt(value, encryption_provider);
+property_jsons.AddMember(rapidjson::Value(name, alloc),
rapidjson::Value(encrypted_value, alloc), alloc);
Review Comment:
Overrides as a name is fine for me, but their expected format needs to be
clearly defined.
I think we should standardize whether the values in the overrides are
already encrypted (if sensitive) or not encrypted. I'm not sure how we
currently decide about encryption based on this control flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2283 Create tool to encrypt sensitive properties in config.yml [nifi-minifi-cpp]
szaszm commented on code in PR #1725:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1725#discussion_r1512869627
##
libminifi/include/core/ProcessGroup.h:
##
@@ -205,6 +205,8 @@ class ProcessGroup : public CoreComponent {
*/
std::shared_ptr
findControllerService(const std::string ) const;
+ std::vector>
getAllControllerServices() const;
Review Comment:
That would be an improvement. Optionally, they could be wrapped in
gsl::not_null. Whether a const refernce is enough for all consumers also needs
to be considered, but if it is, then less unnecessary exposure is better.
Currently it's pointers to mutable objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-10883) SnowflakeComputingConnectionPool fails on Java 21
[
https://issues.apache.org/jira/browse/NIFI-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Turcsanyi resolved NIFI-10883.
Resolution: Information Provided
> SnowflakeComputingConnectionPool fails on Java 21
> -
>
> Key: NIFI-10883
> URL: https://issues.apache.org/jira/browse/NIFI-10883
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
>
> Stack trace:
> {code:java}
> 2022-11-28 12:03:05,585 ERROR [Timer-Driven Process Thread-1]
> o.a.n.p.standard.PutDatabaseRecord
> PutDatabaseRecord[id=c8f95698-77b8-379e-1a39-f4f9fa48bd91] Failed to put
> Records to database for
> StandardFlowFileRecord[uuid=41f457f5-3d38-426c-98e3-63ec75b57548,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1669633380959-1, container=default,
> section=1], offset=0,
> length=25],offset=0,name=41f457f5-3d38-426c-98e3-63ec75b57548,size=25].
> Routing to failure.
> org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException:
> Cannot create PoolableConnectionFactory (isValid() returned false)
> at
> org.apache.nifi.dbcp.AbstractDBCPConnectionPool.getConnection(AbstractDBCPConnectionPool.java:573)
> at
> org.apache.nifi.dbcp.AbstractDBCPConnectionPool.getConnection(AbstractDBCPConnectionPool.java:550)
> at org.apache.nifi.dbcp.DBCPService.getConnection(DBCPService.java:55)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
> at jdk.proxy12/jdk.proxy12.$Proxy88.getConnection(Unknown Source)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1356)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
> at
> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:59)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory
> (isValid() returned false)
> at
> org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:653)
> at
> org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:531)
> at
> org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:731)
> at
> org.apache.nifi.dbcp.AbstractDBCPConnectionPool.getConnection(AbstractDBCPConnectionPool.java:560)
> ... 21 common frames omitted
> Caused by: java.sql.SQLException: isValid() returned false
> at
> org.apache.commons.dbcp2.PoolableConnection.validate(PoolableConnection.java:335)
> at
> org.apache.commons.dbcp2.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:653)
> at
> org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:109)
> at
> org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:649)
> ... 24 common frames omitted
> {code}
> Environment:
> - compiled with 1.8.0_312-b07 (Temurin)
> -- run on 1.8.0_312-b07 (Temurin) => no error
> -- run on Temurin-11.0.17+8 => no error
> -- run on Temurin-17.0.5+8 => failure
> - compiled with Temurin-17.0.5+8
> -- run on Temurin-17.0.5+8 => failure
--
This
[jira] [Updated] (NIFI-12760) [MiNiFi] MiNiFi toolkit encrypt config should support encrypting flow.json.raw file
[ https://issues.apache.org/jira/browse/NIFI-12760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdei updated NIFI-12760: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > [MiNiFi] MiNiFi toolkit encrypt config should support encrypting > flow.json.raw file > --- > > Key: NIFI-12760 > URL: https://issues.apache.org/jira/browse/NIFI-12760 > Project: Apache NiFi > Issue Type: Improvement > Components: MiNiFi >Reporter: Ferenc Erdei >Assignee: Ferenc Kis >Priority: Minor > Labels: minifi-java > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > MiNiFi flows can be encrypted when downloading from ingestors or through c2 > protocol, but when we edit it manually there is no way to encrypt sensitive > properties. > The nifi-toolkit-encrypt config supports encrypting the flow.json.gz, but it > works only for this format. > MiNiFi should provide a similar way to encrypt flows manually. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978792542 @tpalfy You may be right. I just tested the refresh mechanism (not did not wait the 3600 s until the first access token really expired) with my intended fix that morning. But the refreshed token nonetheless needs to be considered by the processor accessing the IMAP server. And this happens in your `buildurl method`. So the real fix needs to be a bit different. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12760) [MiNiFi] MiNiFi toolkit encrypt config should support encrypting flow.json.raw file
[ https://issues.apache.org/jira/browse/NIFI-12760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823619#comment-17823619 ] ASF subversion and git services commented on NIFI-12760: Commit d1b6aab9d36e713adb377f6fec14ea376d980fbe in nifi's branch refs/heads/main from Ferenc Kis [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d1b6aab9d3 ] NIFI-12760 Flow sensitive properties encryption support in toolkit Signed-off-by: Ferenc Erdei This closes #8430. > [MiNiFi] MiNiFi toolkit encrypt config should support encrypting > flow.json.raw file > --- > > Key: NIFI-12760 > URL: https://issues.apache.org/jira/browse/NIFI-12760 > Project: Apache NiFi > Issue Type: Improvement > Components: MiNiFi >Reporter: Ferenc Erdei >Assignee: Ferenc Kis >Priority: Minor > Labels: minifi-java > Time Spent: 40m > Remaining Estimate: 0h > > MiNiFi flows can be encrypted when downloading from ingestors or through c2 > protocol, but when we edit it manually there is no way to encrypt sensitive > properties. > The nifi-toolkit-encrypt config supports encrypting the flow.json.gz, but it > works only for this format. > MiNiFi should provide a similar way to encrypt flows manually. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12760 Flow sensitive properties encryption support in toolkit [nifi]
ferencerdei closed pull request #8430: NIFI-12760 Flow sensitive properties encryption support in toolkit URL: https://github.com/apache/nifi/pull/8430 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-10883) SnowflakeComputingConnectionPool fails on Java 21
[
https://issues.apache.org/jira/browse/NIFI-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823618#comment-17823618
]
Peter Turcsanyi commented on NIFI-10883:
[SNOW-917458|https://github.com/snowflakedb/snowflake-jdbc/issues/1512] has
been solved in snowflake-jdbc:3.14.4 and NiFi 2.0 will use the fixed version
(NIFI-12497).
If SnowflakeComputingConnectionPool is used with ExecuteSQL[Record] (and also
in other cases when InaccessibleObjectException occurs), then the following
entry needs to be added in bootstrap.conf:
{code:java}
java.arg.add-opens.java.nio=--add-opens=java.base/java.nio=ALL-UNNAMED
{code}
As nifi-snowflake nars are not part of the default distribution bundle of NiFi,
bootstrap.conf does not contain the setting by default. It needs to be added on
demand.
> SnowflakeComputingConnectionPool fails on Java 21
> -
>
> Key: NIFI-10883
> URL: https://issues.apache.org/jira/browse/NIFI-10883
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
>
> Stack trace:
> {code:java}
> 2022-11-28 12:03:05,585 ERROR [Timer-Driven Process Thread-1]
> o.a.n.p.standard.PutDatabaseRecord
> PutDatabaseRecord[id=c8f95698-77b8-379e-1a39-f4f9fa48bd91] Failed to put
> Records to database for
> StandardFlowFileRecord[uuid=41f457f5-3d38-426c-98e3-63ec75b57548,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1669633380959-1, container=default,
> section=1], offset=0,
> length=25],offset=0,name=41f457f5-3d38-426c-98e3-63ec75b57548,size=25].
> Routing to failure.
> org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException:
> Cannot create PoolableConnectionFactory (isValid() returned false)
> at
> org.apache.nifi.dbcp.AbstractDBCPConnectionPool.getConnection(AbstractDBCPConnectionPool.java:573)
> at
> org.apache.nifi.dbcp.AbstractDBCPConnectionPool.getConnection(AbstractDBCPConnectionPool.java:550)
> at org.apache.nifi.dbcp.DBCPService.getConnection(DBCPService.java:55)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
> at jdk.proxy12/jdk.proxy12.$Proxy88.getConnection(Unknown Source)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1356)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
> at
> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:59)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory
> (isValid() returned false)
> at
> org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:653)
> at
> org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:531)
> at
> org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:731)
> at
> org.apache.nifi.dbcp.AbstractDBCPConnectionPool.getConnection(AbstractDBCPConnectionPool.java:560)
> ... 21 common frames omitted
> Caused by: java.sql.SQLException: isValid() returned false
> at
> org.apache.commons.dbcp2.PoolableConnection.validate(PoolableConnection.java:335)
> at
>
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
tpalfy commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978692871 Thanks @AnTu2702 and @RomanOttGmx for the explanaion. As for your fix @AnTu2702 I'm a bit confused though. If we only refresh the token but don't rebuild the url I don't see how that would work. Yes, we have a fresh token but are we actually using it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12846) AWS Assume Role Credentials with VPCE Endpoint URL cannot handle the Region
[
https://issues.apache.org/jira/browse/NIFI-12846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Turcsanyi updated NIFI-12846:
---
Status: Patch Available (was: Open)
> AWS Assume Role Credentials with VPCE Endpoint URL cannot handle the Region
>
>
> Key: NIFI-12846
> URL: https://issues.apache.org/jira/browse/NIFI-12846
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.25.0
>Reporter: Peter Turcsanyi
>Assignee: Peter Turcsanyi
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case of custom Endpoint URLs, the AWS client library may be able to parse
> the Region from the URL but can not handle VPCE URLs (e.g.
> [https://vpce-{*}-{*}***-eu-central-1a.sts.eu-central-1.vpce.amazonaws.com).|https://vpce-%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A%2A-%2A%2A%2A%2A%2A%2A%2A%2A-eu-central-1a.sts.eu-central-1.vpce.amazonaws.com)./]
> {code:java}
> 2024-02-27 13:13:04,102 ERROR [Timer-Driven Process Thread-1]
> o.apache.nifi.processors.aws.s3.ListS3
> ListS3[id=d5e08c19-a155-3b34-e9e6-dbd70e048cd1] Failed to list contents of
> bucket
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> Credential should be scoped to a valid region. (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code: SignatureDoesNotMatch;
> Request ID: 7820b219-dee5-4989-8d0c-46523
> 1469705; Proxy: null)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1731)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1698)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1687)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:532)
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:501)
> ... {code}
> Use the explicit Region property (added in NIFI-10791) for VPCE endpoints.
> The issue affects the 1.x line only. In NiFi 2.0, the AWS processors
> (including the credential handling) have been refactored (NIFI-12144).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12846 Fixed Region handling for AWS Assume Role Credentials with… [nifi]
turcsanyip opened a new pull request, #8473: URL: https://github.com/apache/nifi/pull/8473 … VPCE Endpoint URL # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978453332 I was able to build and deploy a local nar file...  ...working fine with the patch! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-2311) Initial support for recordset readers and writers
Martin Zink created MINIFICPP-2311: -- Summary: Initial support for recordset readers and writers Key: MINIFICPP-2311 URL: https://issues.apache.org/jira/browse/MINIFICPP-2311 Project: Apache NiFi MiNiFi C++ Issue Type: New Feature Reporter: Martin Zink Assignee: Martin Zink -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] MINIFICPP-2282 Support re-encryption of sensitive properties [nifi-minifi-cpp]
fgerlits commented on code in PR #1739:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1739#discussion_r1512504892
##
encrypt-config/FlowConfigEncryptor.cpp:
##
@@ -61,114 +62,150 @@ std::vector
listSensitiveProperties(const minifi::core::Proce
std::vector processors;
process_group.getAllProcessors(processors);
- for (const auto *processor : processors) {
+ for (const auto* processor : processors) {
gsl_Expects(processor);
for (const auto& [_, property] : processor->getProperties()) {
if (property.isSensitive()) {
sensitive_properties.push_back(SensitiveProperty{
-.type = Type::Processor,
+.component_type = ComponentType::Processor,
.component_id = processor->getUUID(),
.component_name = processor->getName(),
.property_name = property.getName(),
-.property_display_name = property.getDisplayName()});
+.property_display_name = property.getDisplayName(),
+.property_value = property.getValue().to_string()});
}
}
}
- for (const auto _service_node :
process_group.getAllControllerServices()) {
+ for (const auto& controller_service_node :
process_group.getAllControllerServices()) {
gsl_Expects(controller_service_node);
const auto controller_service =
controller_service_node->getControllerServiceImplementation();
gsl_Expects(controller_service);
for (const auto& [_, property] : controller_service->getProperties()) {
if (property.isSensitive()) {
sensitive_properties.push_back(SensitiveProperty{
-.type = Type::ControllerService,
+.component_type = ComponentType::ControllerService,
.component_id = controller_service->getUUID(),
.component_name = controller_service->getName(),
.property_name = property.getName(),
-.property_display_name = property.getDisplayName()});
+.property_display_name = property.getDisplayName(),
+.property_value = property.getValue().to_string()});
}
}
}
return sensitive_properties;
}
-template
-void encryptSensitiveValuesInFlowConfigImpl(
-const minifi::encrypt_config::EncryptionKeys& keys, const
std::filesystem::path& minifi_home, const std::filesystem::path&
flow_config_path, Func create_overrides) {
- const auto configure = std::make_shared();
+std::unordered_map> createOverridesInteractively(const
std::vector& sensitive_properties) {
+ std::unordered_map> overrides;
+ std::cout << '\n';
+ for (const auto& sensitive_property : sensitive_properties) {
+std::cout << magic_enum::enum_name(sensitive_property.component_type) << "
" << sensitive_property.component_name << " (" <<
sensitive_property.component_id.to_string() << ") "
+ << "has sensitive property " <<
sensitive_property.property_display_name << "\nenter a new value or press
Enter to keep the current value unchanged: ";
+std::cout.flush();
+std::string new_value;
+std::getline(std::cin, new_value);
+if (!new_value.empty()) {
+
overrides[sensitive_property.component_id].emplace(sensitive_property.property_name,
new_value);
+}
+ }
+ return overrides;
+}
+
+std::unordered_map> createOverridesForSingleProperty(
+const std::vector& sensitive_properties, const
std::string& component_id, const std::string& property_name, const std::string&
property_value) {
+ const auto sensitive_property_it =
std::ranges::find_if(sensitive_properties, [&](const auto& sensitive_property) {
+return sensitive_property.component_id.to_string().view() == component_id
&& (sensitive_property.property_name == property_name ||
sensitive_property.property_display_name == property_name);
+ });
+ if (sensitive_property_it == sensitive_properties.end()) {
+std::cout << "No sensitive property found with this component ID and
property name.\n";
+return {};
+ }
+ return {{sensitive_property_it->component_id,
{{sensitive_property_it->property_name, property_value;
+}
+
+std::unordered_map> createOverridesForReEncryption(const
std::vector& sensitive_properties) {
+ std::unordered_map> overrides;
+ for (const auto& sensitive_property : sensitive_properties) {
+
overrides[sensitive_property.component_id].emplace(sensitive_property.property_name,
sensitive_property.property_value);
+ }
+ return overrides;
+}
+
+} // namespace
+
+namespace org::apache::nifi::minifi::encrypt_config::flow_config_encryptor {
+
+EncryptionRequest::EncryptionRequest(EncryptionType type) : type{type} {
+ gsl_Expects(type == EncryptionType::Interactive || type ==
EncryptionType::ReEncrypt);
+}
+
+EncryptionRequest::EncryptionRequest(std::string_view component_id,
std::string_view property_name, std::string_view property_value)
+: type{EncryptionType::SingleProperty},
+ component_id{component_id},
+ property_name{property_name},
+
Re: [PR] MINIFICPP-2283 Create tool to encrypt sensitive properties in config.yml [nifi-minifi-cpp]
fgerlits commented on code in PR #1725:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1725#discussion_r1512455086
##
libminifi/src/core/json/JsonFlowSerializer.cpp:
##
@@ -47,14 +47,21 @@ void
JsonFlowSerializer::encryptSensitiveProperties(rapidjson::Value _j
}
if (properties.at(name).isSensitive()) {
auto& value = property.value;
- const std::string_view value_sv{value.GetString(),
value.GetStringLength()};
+ const std::string_view value_sv = component_overrides.contains(name) ?
component_overrides.at(name) : std::string_view{value.GetString(),
value.GetStringLength()};
const std::string encrypted_value =
utils::crypto::property_encryption::encrypt(value_sv, encryption_provider);
value.SetString(encrypted_value.c_str(), encrypted_value.size(), alloc);
}
+component_overrides.erase(name);
+ }
+
+ for (const auto& [name, value] : component_overrides) {
+const std::string encrypted_value =
utils::crypto::property_encryption::encrypt(value, encryption_provider);
+property_jsons.AddMember(rapidjson::Value(name, alloc),
rapidjson::Value(encrypted_value, alloc), alloc);
Review Comment:
Also, this part of the code handles the case when the property given by the
user to the `encrypt-config` tool does not exist in the JSON, yet. In this
case, we add the property to the JSON flow configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2283 Create tool to encrypt sensitive properties in config.yml [nifi-minifi-cpp]
fgerlits commented on code in PR #1725:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1725#discussion_r1512449471
##
libminifi/src/core/json/JsonFlowSerializer.cpp:
##
@@ -47,14 +47,21 @@ void
JsonFlowSerializer::encryptSensitiveProperties(rapidjson::Value _j
}
if (properties.at(name).isSensitive()) {
auto& value = property.value;
- const std::string_view value_sv{value.GetString(),
value.GetStringLength()};
+ const std::string_view value_sv = component_overrides.contains(name) ?
component_overrides.at(name) : std::string_view{value.GetString(),
value.GetStringLength()};
const std::string encrypted_value =
utils::crypto::property_encryption::encrypt(value_sv, encryption_provider);
value.SetString(encrypted_value.c_str(), encrypted_value.size(), alloc);
}
+component_overrides.erase(name);
+ }
+
+ for (const auto& [name, value] : component_overrides) {
+const std::string encrypted_value =
utils::crypto::property_encryption::encrypt(value, encryption_provider);
+property_jsons.AddMember(rapidjson::Value(name, alloc),
rapidjson::Value(encrypted_value, alloc), alloc);
Review Comment:
Encrypted or not is a red herring. The property value in the JSON (or YAML)
may or may not be already encrypted; if it isn't, then it will be encrypted
now. Overrides are used by the `encrypt-config` tool to replace the existing
property value with a new value given by the user. "Override" may not be the
best name for this; do you have a suggestion for a better name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-2283 Create tool to encrypt sensitive properties in config.yml [nifi-minifi-cpp]
fgerlits commented on code in PR #1725:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1725#discussion_r1512442049
##
libminifi/include/core/ProcessGroup.h:
##
@@ -205,6 +205,8 @@ class ProcessGroup : public CoreComponent {
*/
std::shared_ptr
findControllerService(const std::string ) const;
+ std::vector>
getAllControllerServices() const;
Review Comment:
do you mean the vector should contain `const ControllerServiceNode*`s?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978198291 > @AnTu2702 The `oauth2AccessTokenProviderOptional` field is used in the `buildUrl` method where we call `oauth2AccessTokenProvider.getAccessDetails().getAccessToken()`. The `getAccessDetail()` should take care of refreshing/reacquiring tokens (this is the same call you suggest doing at the beginning of `onTrigger`). > > So according to this, token expiration is handled already. Do we have proof that this doesn't work for some reason? Hey. The `this.messageReceiver = this.buildMessageReceiver(context)` and thus `buildUrl` method in line 331 is only called once. Reason is line 329: `if (this.messageReceiver == null)`  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12859) OAUTH2 Token Refresh not working in AbstractEmailProcessor
[ https://issues.apache.org/jira/browse/NIFI-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823484#comment-17823484 ] Andreas Tully commented on NIFI-12859: -- There is already an ongoing discussion about this here: [https://github.com/apache/nifi/pull/6900] > OAUTH2 Token Refresh not working in AbstractEmailProcessor > --- > > Key: NIFI-12859 > URL: https://issues.apache.org/jira/browse/NIFI-12859 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.24.0, 1.25.0 >Reporter: Roman >Priority: Major > Labels: OAuth2 > Attachments: image-2024-03-04-11-00-25-665.png > > > The ConsumeIMAP Processor implements the AbstractEmailProcessor. > With Issue NIFI-11107 there was introduced OAUTH2 Support. Initial OAUTH is > working fine. > But the refresh token implementation is not working with Credential Flow in > OAUTH Processor. > Reason: > Method to refresh token => oauth2AccessTokenProvider.getAccessDetails(); is > called only in onScheduled Method of AbstractEmailProcessor. > But once the ConsumeIMAPProcessor is startet (also for cron based scheduling) > only the onTriggeredMethod will be triggered. > onScheduled will be only triggered on start 1time. > onTriggered will be also triggered after start of the AbstractEmailProcessor. > Abstract-Email-Processor: > [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/AbstractEmailProcessor.java] > Here you find details: > [https://github.com/apache/nifi/pull/6900] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-12859) OAUTH2 Token Refresh not working in AbstractEmailProcessor
[ https://issues.apache.org/jira/browse/NIFI-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823131#comment-17823131 ] Andreas Tully edited comment on NIFI-12859 at 3/5/24 8:14 AM: -- I debugged the AbstractEmailProcessor.java and it seems, that AbstractEmailProcessor.java does not handle token expiration at all. The StandardOAuth2AccessTokenProvider.java does, but it would need the AbstractEmailProcessor.java to call the: oauth2AccessTokenProvider.getAccessDetails() in the onTrigger() method instead the onSchedule() method to do so. The AbstractEmailProcessor.java should be fixed to handle expired OAuth2 access tokens! was (Author: JIRAUSER304486): I debugged the AbstractEmailProcessor.java and it seems, that AbstractEmailProcessor.java does not handle token expiration at all. The StandardOAuth2AccessTokenProvider.java does, but it would need the AbstractEmailProcessor.java to call the: oauth2AccessTokenProvider.getAccessDetails() in the onTrigger() method instead the onSchedule() method to do so. [!https://private-user-images.githubusercontent.com/5054213/309678670-40403a5b-3a2f-4174-a1a2-af204650bd0e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDk1NTEwOTcsIm5iZiI6MTcwOTU1MDc5NywicGF0aCI6Ii81MDU0MjEzLzMwOTY3ODY3MC00MDQwM2E1Yi0zYTJmLTQxNzQtYTFhMi1hZjIwNDY1MGJkMGUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDMwNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAzMDRUMTExMzE3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZDgwNmZiMDc4ZTNhZWYzOWI3YzE4NmNlZjE0MWI3ZmFlZmU0OWIxODdlMGE0M2JhN2EwYzc4NWJhMDk2MGJlMSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.tFkjC7_5Bts96mIVPHYhSx61FC2cVwMK_gH9R8QDJUA!|https://private-user-images.githubusercontent.com/5054213/309678670-40403a5b-3a2f-4174-a1a2-af204650bd0e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDk1NTEwOTcsIm5iZiI6MTcwOTU1MDc5NywicGF0aCI6Ii81MDU0MjEzLzMwOTY3ODY3MC00MDQwM2E1Yi0zYTJmLTQxNzQtYTFhMi1hZjIwNDY1MGJkMGUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDMwNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAzMDRUMTExMzE3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZDgwNmZiMDc4ZTNhZWYzOWI3YzE4NmNlZjE0MWI3ZmFlZmU0OWIxODdlMGE0M2JhN2EwYzc4NWJhMDk2MGJlMSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.tFkjC7_5Bts96mIVPHYhSx61FC2cVwMK_gH9R8QDJUA] The AbstractEmailProcessor.java should be fixed to handle expired OAuth2 access tokens! > OAUTH2 Token Refresh not working in AbstractEmailProcessor > --- > > Key: NIFI-12859 > URL: https://issues.apache.org/jira/browse/NIFI-12859 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.24.0, 1.25.0 >Reporter: Roman >Priority: Major > Labels: OAuth2 > Attachments: image-2024-03-04-11-00-25-665.png > > > The ConsumeIMAP Processor implements the AbstractEmailProcessor. > With Issue NIFI-11107 there was introduced OAUTH2 Support. Initial OAUTH is > working fine. > But the refresh token implementation is not working with Credential Flow in > OAUTH Processor. > Reason: > Method to refresh token => oauth2AccessTokenProvider.getAccessDetails(); is > called only in onScheduled Method of AbstractEmailProcessor. > But once the ConsumeIMAPProcessor is startet (also for cron based scheduling) > only the onTriggeredMethod will be triggered. > onScheduled will be only triggered on start 1time. > onTriggered will be also triggered after start of the AbstractEmailProcessor. > Abstract-Email-Processor: > [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/AbstractEmailProcessor.java] > Here you find details: > [https://github.com/apache/nifi/pull/6900] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978169106 Overview:  ConsumeImap:   StandardOAuth2TokenProvider:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
