[jira] [Updated] (NIFI-11241) Introduce API for building Processors in Python
[ https://issues.apache.org/jira/browse/NIFI-11241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11241: Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Introduce API for building Processors in Python > --- > > Key: NIFI-11241 > URL: https://issues.apache.org/jira/browse/NIFI-11241 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework, Documentation & Website, Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 2.0.0 > > Time Spent: 8.5h > Remaining Estimate: 0h > > Introduce an API for building Processors in Python. Use Py4J for > communication layer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11241) Introduce API for building Processors in Python
[ https://issues.apache.org/jira/browse/NIFI-11241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712518#comment-17712518 ] ASF subversion and git services commented on NIFI-11241: Commit 7f7cccfdc2f0788f5fc26703ac8e836092d6d06b in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7f7cccfdc2 ] NIFI-11241 Initial implementation of Python-based Processor API with Py4J - Updated GitHub workflow so that system tests include Python 3.9 - Updated GitHub actions to build necessary modules for system tests This closes #7003 Co-authored-by: David Handermann Signed-off-by: David Handermann > Introduce API for building Processors in Python > --- > > Key: NIFI-11241 > URL: https://issues.apache.org/jira/browse/NIFI-11241 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework, Documentation & Website, Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > Introduce an API for building Processors in Python. Use Py4J for > communication layer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory closed pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
exceptionfactory closed pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with… URL: https://github.com/apache/nifi/pull/7003 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mr1716 closed pull request #7162: NIFI-11432 Upgrade Angular Version to 1.8.3
mr1716 closed pull request #7162: NIFI-11432 Upgrade Angular Version to 1.8.3 URL: https://github.com/apache/nifi/pull/7162 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] scottyaslan commented on a diff in pull request #7162: NIFI-11432 Upgrade Angular Version to 1.8.3
scottyaslan commented on code in PR #7162: URL: https://github.com/apache/nifi/pull/7162#discussion_r1167154119 ## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/frontend/package.json: ## @@ -20,7 +20,7 @@ "preinstall": "npm --ignore-scripts install npm-force-resolutions && npm-force-resolutions" }, "dependencies": { -"angular": "1.8.2", +"angular": "1.8.3", "angular-animate": "1.8.2", Review Comment: FYI, these other angular-* packages should also be updated when upgrading angular. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] scottyaslan commented on a diff in pull request #7162: NIFI-11432 Upgrade Angular Version to 1.8.3
scottyaslan commented on code in PR #7162: URL: https://github.com/apache/nifi/pull/7162#discussion_r1163239177 ## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/frontend/package-lock.json: ## @@ -7,7 +7,7 @@ "name": "apache-nifi", "hasInstallScript": true, "dependencies": { -"angular": "1.8.2", +"angular": "1.8.3", Review Comment: This is not correct. Anytime the dependencies are changed in the package.json the package-lock.json should be deleted and regenerated with the appropriate version of npm (v8.1.2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-11440) Speed up Hive Metastore-based tests for Hive 3 and Iceberg
[ https://issues.apache.org/jira/browse/NIFI-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-11440. - Fix Version/s: 2.0.0 1.22.0 Resolution: Fixed > Speed up Hive Metastore-based tests for Hive 3 and Iceberg > -- > > Key: NIFI-11440 > URL: https://issues.apache.org/jira/browse/NIFI-11440 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Bathori >Assignee: Mark Bathori >Priority: Major > Fix For: 2.0.0, 1.22.0 > > Time Spent: 1h > Remaining Estimate: 0h > > The Hive Metastore based unit tests take a lot of time because the metastore > is initialized before every test. These executions can be speed up by > changing the initialization to be called only once before starting the tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11382) UI - Processor property configurations throws console error when referenced parameter is deleted
[ https://issues.apache.org/jira/browse/NIFI-11382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFI-11382: --- Description: When a processor is configured with a referenced parameter and that parameter is deleted, the next time the user configures the property, it will throw a console error. This is related to NIFI-11451 was:When a processor is configured with a referenced parameter and that parameter is deleted, the next time the user configures the property, it will throw a console error. > UI - Processor property configurations throws console error when referenced > parameter is deleted > > > Key: NIFI-11382 > URL: https://issues.apache.org/jira/browse/NIFI-11382 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.20.0 >Reporter: Margot Tien >Assignee: Margot Tien >Priority: Minor > Attachments: Screen Shot 2023-03-31 at 12.36.32 PM.png > > > When a processor is configured with a referenced parameter and that parameter > is deleted, the next time the user configures the property, it will throw a > console error. > > This is related to NIFI-11451 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11451) UI - property table console error
[ https://issues.apache.org/jira/browse/NIFI-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFI-11451: --- Description: When attempting to reference a parameter or create a controller service inline there is a console error in devTools: This is related to NIFI-11382 was: When attempting to reference a parameter or create a controller service inline there is a console error in devTools: > UI - property table console error > - > > Key: NIFI-11451 > URL: https://issues.apache.org/jira/browse/NIFI-11451 > Project: Apache NiFi > Issue Type: Bug >Reporter: Scott Aslan >Priority: Major > Attachments: Screenshot 2023-04-14 at 1.10.58 PM.png > > > When attempting to reference a parameter or create a controller service > inline there is a console error in devTools: > > This is related to NIFI-11382 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11440) Speed up Hive Metastore-based tests for Hive 3 and Iceberg
[ https://issues.apache.org/jira/browse/NIFI-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11440: Summary: Speed up Hive Metastore-based tests for Hive 3 and Iceberg (was: Speed up Hive Metastore based unit tests) > Speed up Hive Metastore-based tests for Hive 3 and Iceberg > -- > > Key: NIFI-11440 > URL: https://issues.apache.org/jira/browse/NIFI-11440 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Bathori >Assignee: Mark Bathori >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > The Hive Metastore based unit tests take a lot of time because the metastore > is initialized before every test. These executions can be speed up by > changing the initialization to be called only once before starting the tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11440) Speed up Hive Metastore based unit tests
[ https://issues.apache.org/jira/browse/NIFI-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712488#comment-17712488 ] ASF subversion and git services commented on NIFI-11440: Commit 15a1134c85c38e1d6fe146259ca8b2f77c6cdbf5 in nifi's branch refs/heads/support/nifi-1.x from Mark Bathori [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=15a1134c85 ] NIFI-11440 Speed up Iceberg Hive Metastore Tests This closes #7170 Signed-off-by: David Handermann (cherry picked from commit 44a7f7f38b3950a8a0135d1ce8602a4890c3d995) > Speed up Hive Metastore based unit tests > > > Key: NIFI-11440 > URL: https://issues.apache.org/jira/browse/NIFI-11440 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Bathori >Assignee: Mark Bathori >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > The Hive Metastore based unit tests take a lot of time because the metastore > is initialized before every test. These executions can be speed up by > changing the initialization to be called only once before starting the tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11440) Speed up Hive Metastore based unit tests
[ https://issues.apache.org/jira/browse/NIFI-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712487#comment-17712487 ] ASF subversion and git services commented on NIFI-11440: Commit 44a7f7f38b3950a8a0135d1ce8602a4890c3d995 in nifi's branch refs/heads/main from Mark Bathori [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=44a7f7f38b ] NIFI-11440 Speed up Iceberg Hive Metastore Tests This closes #7170 Signed-off-by: David Handermann > Speed up Hive Metastore based unit tests > > > Key: NIFI-11440 > URL: https://issues.apache.org/jira/browse/NIFI-11440 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Bathori >Assignee: Mark Bathori >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > The Hive Metastore based unit tests take a lot of time because the metastore > is initialized before every test. These executions can be speed up by > changing the initialization to be called only once before starting the tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory closed pull request #7170: NIFI-11440: Speed up Hive Metastore based unit tests
exceptionfactory closed pull request #7170: NIFI-11440: Speed up Hive Metastore based unit tests URL: https://github.com/apache/nifi/pull/7170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] scottyaslan opened a new pull request, #7174: [NIFIF-11433] update angular, d3, moment, slickgrid, and jquery depen…
scottyaslan opened a new pull request, #7174: URL: https://github.com/apache/nifi/pull/7174 …dencies # Summary [NIFI-11433](https://issues.apache.org/jira/browse/NIFI-11433) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 11 - [ ] JDK 17 ### Licensing - [X] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [X] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [X] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-11451) UI - property table console error
Scott Aslan created NIFI-11451: -- Summary: UI - property table console error Key: NIFI-11451 URL: https://issues.apache.org/jira/browse/NIFI-11451 Project: Apache NiFi Issue Type: Bug Reporter: Scott Aslan Attachments: Screenshot 2023-04-14 at 1.10.58 PM.png When attempting to reference a parameter or create a controller service inline there is a console error in devTools: -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11433) Upgrade moment to 2.29.4
[ https://issues.apache.org/jira/browse/NIFI-11433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFI-11433: --- Description: Upgrade d3 to 7.6.1 to be able to remove d3-selection-multi. Upgrade Angular and related packages to 1.8.3 Upgrade to jQuery 3.6.4 Upgrade to slickgrid 2.4.45 Upgrade moment to 2.29.4 to remediate vulnerability [Regular Expression Denial of Service (ReDoS) in moment | CVE-2022-31129 | Snyk|https://security.snyk.io/vuln/SNYK-JS-MOMENT-2944238] was:Upgrade moment to 2.29.4 to remediate vulnerability [Regular Expression Denial of Service (ReDoS) in moment | CVE-2022-31129 | Snyk|https://security.snyk.io/vuln/SNYK-JS-MOMENT-2944238] > Upgrade moment to 2.29.4 > > > Key: NIFI-11433 > URL: https://issues.apache.org/jira/browse/NIFI-11433 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.21.0 >Reporter: Mike R >Assignee: Scott Aslan >Priority: Major > > Upgrade d3 to 7.6.1 to be able to remove d3-selection-multi. > Upgrade Angular and related packages to 1.8.3 > Upgrade to jQuery 3.6.4 > Upgrade to slickgrid 2.4.45 > Upgrade moment to 2.29.4 to remediate vulnerability [Regular Expression > Denial of Service (ReDoS) in moment | CVE-2022-31129 | > Snyk|https://security.snyk.io/vuln/SNYK-JS-MOMENT-2944238] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-2104) CWEL Output Format validation
Marton Szasz created MINIFICPP-2104: --- Summary: CWEL Output Format validation Key: MINIFICPP-2104 URL: https://issues.apache.org/jira/browse/MINIFICPP-2104 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Marton Szasz When using an invalid output format, like 'Text' instead of 'Plaintext', CWEL prints no errors, runs normally, but doesn't output anything. We should improve property validation to give an error message, instead of silently failing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11450) Node respecting nifi.flowcontroller.autoResumeState when joining cluster
[ https://issues.apache.org/jira/browse/NIFI-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Lord updated NIFI-11450: - Environment: Nifi version 1.19.1. Linux. JDK11+ (was: Nifi version 11. Linux. JDK11+) > Node respecting nifi.flowcontroller.autoResumeState when joining cluster > > > Key: NIFI-11450 > URL: https://issues.apache.org/jira/browse/NIFI-11450 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.19.1 > Environment: Nifi version 1.19.1. Linux. JDK11+ >Reporter: Phil Lord >Priority: Critical > > Mistakenly tried to connect a node to an active cluster that had the > following setting: nifi.flowcontroller.autoResumeState=true . > The flow.xml/json were both removed completely prior to attempting to connect > node to the cluster, then started the node. The node connected to the > cluster without issue, however all components/controllerServices on that node > were in a stopped state(due to the property mentioned above). > Is this correct behavior given that the flow that was inherited from the > cluster had components started/etc. I would think the node should either > fail to join the cluster/start? > This was running Nifi 1.19.1... unclear if it's resolved in latest release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11450) Node respecting nifi.flowcontroller.autoResumeState when joining cluster
Phil Lord created NIFI-11450: Summary: Node respecting nifi.flowcontroller.autoResumeState when joining cluster Key: NIFI-11450 URL: https://issues.apache.org/jira/browse/NIFI-11450 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.19.1 Environment: Nifi version 11. Linux. JDK11+ Reporter: Phil Lord Mistakenly tried to connect a node to an active cluster that had the following setting: nifi.flowcontroller.autoResumeState=true . The flow.xml/json were both removed completely prior to attempting to connect node to the cluster, then started the node. The node connected to the cluster without issue, however all components/controllerServices on that node were in a stopped state(due to the property mentioned above). Is this correct behavior given that the flow that was inherited from the cluster had components started/etc. I would think the node should either fail to join the cluster/start? This was running Nifi 1.19.1... unclear if it's resolved in latest release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
exceptionfactory commented on code in PR #7003: URL: https://github.com/apache/nifi/pull/7003#discussion_r1167047760 ## nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java: ## @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.py4j; + +import org.apache.nifi.py4j.client.JavaObjectBindings; +import org.apache.nifi.py4j.client.NiFiPythonGateway; +import org.apache.nifi.py4j.client.StandardPythonClient; +import org.apache.nifi.py4j.server.NiFiGatewayServer; +import org.apache.nifi.python.ControllerServiceTypeLookup; +import org.apache.nifi.python.PythonController; +import org.apache.nifi.python.PythonProcessConfig; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import py4j.CallbackClient; +import py4j.GatewayServer; + +import javax.net.ServerSocketFactory; +import javax.net.SocketFactory; +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Collections; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; + +// TODO / Figure Out for MVP: +// MUST DO: +// - Documentation +// - Admin Guide +// - JavaDocs +// - Developer Guide +// - Explain how communication between Java & Python work. +// - Java is preferred, Python is slower and more expensive b/c of network +// - Different Extension Points (FlowFileTransform, RecordTransform) +// - What the API Looks like, Links to JavaDocs for ProcessContext, etc. +// - Example Code +// - Exposing properties +// - Relationships +// - Controller Services +// - Need to update docs to show the interfaces that are exposed, explain how to get these... +// - Design Doc +// - Setup proper logging on the Python side: https://docs.python.org/2/howto/logging-cookbook.html#using-file-rotation +// - For FlowFileTransform, allow the result to contain either a byte array or a String. If a String, just convert in the parent class. +// - Figure out how to deal with Python Packaging +// - Need to figure out how to deal with additionalDetails.html, docs directory in python project typically? +// - Understand how to deal with versioning +// - Look at performance improvements for Py4J - socket comms appear to be INCREDIBLY slow. +// - Create test that calls Python 1M times. Just returns 'hello'. See how long it takes +// - Create test that calls Python 1M times. Returns .toString() and see how long it takes. +// - Will help to understand if it's the call from Java to Python that's slow, Python to Java, or both. +// - Performance concern for TransformRecord +// - Currently, triggering the transform() method is pretty fast. But then the Result object comes back and we have to call into the Python side to call the getters +//over and over. Need to look into instead serializing the entire response as JSON and sending that back. +// - Also, since this is heavy JSON processing, might want to consider ORJSON or something like that instead of inbuilt JSON parser/generator +// - Test pip install nifi-my-proc, does nifi pick it up? +// - When ran DetectObjectInImage with multiple threads, Python died. Need to figure out why. +// - If Python Process dies, need to create a new process and need to then create all of the Processors that were in that Process and initialize them. +//- Milestone 2 or 3, not Milestone 1. +// - Remove test-pypi usage from ExtensionManager.py +// - Additional Interfaces beyond just FlowFileTransform +// - FlowFileSource +// - Restructure Maven projects +// - Should this all go under Framework? +// +// +// CONSIDER: +// - Clustering: Ensure component on all nodes? +// - Consider "pip freeze" type of thing to ensure that python dependencies are same across nodes when joining cluster. +// - Update pytho
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
exceptionfactory commented on code in PR #7003: URL: https://github.com/apache/nifi/pull/7003#discussion_r1167045800 ## nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java: ## @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.py4j; + +import org.apache.nifi.py4j.client.JavaObjectBindings; +import org.apache.nifi.py4j.client.NiFiPythonGateway; +import org.apache.nifi.py4j.client.StandardPythonClient; +import org.apache.nifi.py4j.server.NiFiGatewayServer; +import org.apache.nifi.python.ControllerServiceTypeLookup; +import org.apache.nifi.python.PythonController; +import org.apache.nifi.python.PythonProcessConfig; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import py4j.CallbackClient; +import py4j.GatewayServer; + +import javax.net.ServerSocketFactory; +import javax.net.SocketFactory; +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Collections; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; + +// TODO / Figure Out for MVP: +// MUST DO: +// - Documentation +// - Admin Guide +// - JavaDocs +// - Developer Guide +// - Explain how communication between Java & Python work. +// - Java is preferred, Python is slower and more expensive b/c of network +// - Different Extension Points (FlowFileTransform, RecordTransform) +// - What the API Looks like, Links to JavaDocs for ProcessContext, etc. +// - Example Code +// - Exposing properties +// - Relationships +// - Controller Services +// - Need to update docs to show the interfaces that are exposed, explain how to get these... +// - Design Doc +// - Setup proper logging on the Python side: https://docs.python.org/2/howto/logging-cookbook.html#using-file-rotation +// - For FlowFileTransform, allow the result to contain either a byte array or a String. If a String, just convert in the parent class. +// - Figure out how to deal with Python Packaging +// - Need to figure out how to deal with additionalDetails.html, docs directory in python project typically? +// - Understand how to deal with versioning +// - Look at performance improvements for Py4J - socket comms appear to be INCREDIBLY slow. +// - Create test that calls Python 1M times. Just returns 'hello'. See how long it takes +// - Create test that calls Python 1M times. Returns .toString() and see how long it takes. +// - Will help to understand if it's the call from Java to Python that's slow, Python to Java, or both. +// - Performance concern for TransformRecord +// - Currently, triggering the transform() method is pretty fast. But then the Result object comes back and we have to call into the Python side to call the getters +//over and over. Need to look into instead serializing the entire response as JSON and sending that back. +// - Also, since this is heavy JSON processing, might want to consider ORJSON or something like that instead of inbuilt JSON parser/generator +// - Test pip install nifi-my-proc, does nifi pick it up? +// - When ran DetectObjectInImage with multiple threads, Python died. Need to figure out why. +// - If Python Process dies, need to create a new process and need to then create all of the Processors that were in that Process and initialize them. +//- Milestone 2 or 3, not Milestone 1. +// - Remove test-pypi usage from ExtensionManager.py +// - Additional Interfaces beyond just FlowFileTransform +// - FlowFileSource +// - Restructure Maven projects +// - Should this all go under Framework? +// +// +// CONSIDER: +// - Clustering: Ensure component on all nodes? +// - Consider "pip freeze" type of thing to ensure that python dependencies are same across nodes when joining cluster. +// - Update pytho
[jira] [Updated] (MINIFICPP-2084) Reverse DNS timeout test fails transiently
[ https://issues.apache.org/jira/browse/MINIFICPP-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-2084: Resolution: Fixed Status: Resolved (was: Patch Available) > Reverse DNS timeout test fails transiently > -- > > Key: MINIFICPP-2084 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2084 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > It is due to the fact that the timeout uses a separate thread, and it can > happen that the reverse dns lookup finishes before the 0ms timeout even > starts. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2073) Separate docker build from docker tests in CI
[ https://issues.apache.org/jira/browse/MINIFICPP-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-2073: Resolution: Done Status: Resolved (was: Patch Available) > Separate docker build from docker tests in CI > - > > Key: MINIFICPP-2073 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2073 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Minor > Time Spent: 3h 20m > Remaining Estimate: 0h > > Since the docker tests are not as consistent as we would like, it would be > nice if we could rerun just the tests without the need to rebuild the image. > We could use docker save/load image and github upload/download artifact to > achive this capability. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2018) Add ProcessContext::getStateManager to Lua and Python API
[ https://issues.apache.org/jira/browse/MINIFICPP-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-2018: Resolution: Done Status: Resolved (was: Patch Available) > Add ProcessContext::getStateManager to Lua and Python API > - > > Key: MINIFICPP-2018 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2018 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1538: MINIFICPP-2073 Separate docker build from docker tests in CI
szaszm closed pull request #1538: MINIFICPP-2073 Separate docker build from docker tests in CI URL: https://github.com/apache/nifi-minifi-cpp/pull/1538 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1541: MINIFICPP-2084 Fix flaky Reverse DNS timeout test
szaszm closed pull request #1541: MINIFICPP-2084 Fix flaky Reverse DNS timeout test URL: https://github.com/apache/nifi-minifi-cpp/pull/1541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1530: MINIFICPP-2018 Add ProcessContext::getStateManager to Lua/Python
szaszm closed pull request #1530: MINIFICPP-2018 Add ProcessContext::getStateManager to Lua/Python URL: https://github.com/apache/nifi-minifi-cpp/pull/1530 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-2103) JNI extension fails to compile with clang
[ https://issues.apache.org/jira/browse/MINIFICPP-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-2103: --- Summary: JNI extension fails to compile with clang (was: JNI extension fails to compile on macOS) > JNI extension fails to compile with clang > - > > Key: MINIFICPP-2103 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2103 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > > extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function > for call to 'min' > int actual = > static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, > std::min(remaining, buffer_.size(); > > > ^~~~ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on pull request #7170: NIFI-11440: Speed up Hive Metastore based unit tests
exceptionfactory commented on PR #7170: URL: https://github.com/apache/nifi/pull/7170#issuecomment-1508798033 Thanks for making the adjustments @mark-bathori! I started the automated build and will plan on merging pending successful completion on all platforms. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-11439) Add Storage API URL to GCS Processors
[ https://issues.apache.org/jira/browse/NIFI-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712439#comment-17712439 ] ASF subversion and git services commented on NIFI-11439: Commit d762f35345a8486903a0b30d0923e1e8a3377863 in nifi's branch refs/heads/support/nifi-1.x from Paul Grey [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d762f35345 ] NIFI-11439 Corrected GCS Transit URI for custom Storage API URL This closes #7173 Signed-off-by: David Handermann (cherry picked from commit cff7808543716da5ca9f93356d7d762e15fdd143) > Add Storage API URL to GCS Processors > - > > Key: NIFI-11439 > URL: https://issues.apache.org/jira/browse/NIFI-11439 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Minor > Fix For: 2.0.0, 1.22.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > AWS processors allow the override of the base endpoint for API calls to AWS > services [1]. GCS libraries provide analogous capabilities, as documented > here [2]. Expose a GCS processor property to enable GCS endpoint override > behavior. > [1] > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java#L146-L154 > [2] > https://cloud.google.com/storage/docs/request-endpoints#storage-set-client-endpoint-java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11439) Add Storage API URL to GCS Processors
[ https://issues.apache.org/jira/browse/NIFI-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712438#comment-17712438 ] ASF subversion and git services commented on NIFI-11439: Commit cff7808543716da5ca9f93356d7d762e15fdd143 in nifi's branch refs/heads/main from Paul Grey [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=cff7808543 ] NIFI-11439 Corrected GCS Transit URI for custom Storage API URL This closes #7173 Signed-off-by: David Handermann > Add Storage API URL to GCS Processors > - > > Key: NIFI-11439 > URL: https://issues.apache.org/jira/browse/NIFI-11439 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Paul Grey >Assignee: Paul Grey >Priority: Minor > Fix For: 2.0.0, 1.22.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > AWS processors allow the override of the base endpoint for API calls to AWS > services [1]. GCS libraries provide analogous capabilities, as documented > here [2]. Expose a GCS processor property to enable GCS endpoint override > behavior. > [1] > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java#L146-L154 > [2] > https://cloud.google.com/storage/docs/request-endpoints#storage-set-client-endpoint-java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory closed pull request #7173: NIFI-11439 - correct provenance reporting parameter
exceptionfactory closed pull request #7173: NIFI-11439 - correct provenance reporting parameter URL: https://github.com/apache/nifi/pull/7173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-2103) JNI extension fails to compile on macOS
[ https://issues.apache.org/jira/browse/MINIFICPP-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-2103: --- Description: extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function for call to 'min' int actual = static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, std::min(remaining, buffer_.size(); ^~~~ was: extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function for call to 'min' int actual = static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, std::min(remaining, buffer_.size(); > JNI extension fails to compile on macOS > --- > > Key: MINIFICPP-2103 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2103 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > > extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function > for call to 'min' > int actual = > static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, > std::min(remaining, buffer_.size(); > > > ^~~~ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2103) JNI extension fails to compile on macOS
[ https://issues.apache.org/jira/browse/MINIFICPP-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-2103: --- Description: extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function for call to 'min' int actual = static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, std::min(remaining, buffer_.size(); was:2023-04-14T12:43:23.4194930Z In file included from ../../../extensions/jni/ExecuteJavaControllerService.cpp:19: 2023-04-14T12:43:23.4223740Z In file included from ../../../extensions/jni/ExecuteJavaControllerService.h:36: 2023-04-14T12:43:23.4255880Z In file included from ../../../extensions/jni/ClassRegistrar.h:37: 2023-04-14T12:43:23.4318350Z ../../../extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function for call to 'min' 2023-04-14T12:43:23.4380880Z int actual = static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, std::min(remaining, buffer_.size(); 2023-04-14T12:43:23.4422190Z ^~~~ 2023-04-14T12:43:23.4474240Z /Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:39:1: note: candidate template ignored: deduced conflicting types for parameter '_Tp' ('unsigned long long' vs. 'unsigned long') 2023-04-14T12:43:23.4497750Z min(const _Tp& __a, const _Tp& __b) 2023-04-14T12:43:23.4546180Z ^ 2023-04-14T12:43:23.4648060Z /Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:50:1: note: candidate template ignored: could not match 'initializer_list' against 'unsigned long long' 2023-04-14T12:43:23.4652360Z cd /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-build/google/cloud/storage && ccache /usr/local/opt/ccache/bin/ccache /usr/local/opt/ccache/libexec/c++ -DCURL_STATICLIB -DHAS_EXECINFO=1 -DUSE_REAL_ODBC_TEST_DRIVER -D_SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING -D_SILENCE_CXX20_REL_OPS_DEPRECATION_WARNING -I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-src -I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/absl-src -I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/crc32c-src/include -isystem /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-build -isystem /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/nlohmann -isystem /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/curl-install/include -isystem /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/zlib-install/include -isystem /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/libressl-install/include -I/usr/local/opt/flex/include -std=c++20 -DENABLE_JNI -DOPENSSL_SUPPORT -DSODIUM_STATIC=1 -Wall -Wextra -Werror -O3 -DNDEBUG -std=gnu++20 -isysroot /Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -fPIC -Wall -Wextra -Wconversion -Wno-sign-conversion -MD -MT _deps/google-cloud-cpp-build/google/cloud/storage/CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o -MF CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o.d -o CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o -c /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-src/google/cloud/storage/internal/compute_engine_util.cc 2023-04-14T12:43:23.4671030Z min(initializer_list<_Tp> __t, _Compare __comp) 2023-04-14T12:43:23.4771990Z ^ 2023-04-14T12:43:23.4857020Z /Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:59:1: note: candidate function template not viable: requires single argument '__t', but 2 arguments were provided 2023-04-14T12:43:23.4958490Z min(initializer_list<_Tp> __t) 2023-04-14T12:43:23.4995100Z ^ 2023-04-14T12:43:23.5096600Z /Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:30:1: note: candidate function template not viable: requires 3 arguments, but 2 were provided 2023-04-14T12:43:23.5197380Z min(const _Tp& __a, const _Tp& __b, _Compare __comp) 2023-04-14T12:43:23.5298390Z ^ 2023-04-14T12:43:23.5351750Z 1 error generated. 2023-04-14T12:43:23.5453170Z make[2]: *** [extensions/jni/CMakeFiles/minifi-jni.dir/ExecuteJavaControllerService.cpp.o] Error 1 2023-04-14T12:43:23.350Z make[1]: *** [extensions/jni/CMakeFiles/minifi-jni.dir/all] Error 2 2023-04-14T12:43:23.5656530Z make[1]: *** Waiting for unfinished jobs > JNI extension fails to compile on macOS > --- > > Key: MINIFICPP-2103 >
[jira] [Created] (MINIFICPP-2103) JNI extension fails to compile on macOS
Martin Zink created MINIFICPP-2103: -- Summary: JNI extension fails to compile on macOS Key: MINIFICPP-2103 URL: https://issues.apache.org/jira/browse/MINIFICPP-2103 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Reporter: Martin Zink Assignee: Martin Zink 2023-04-14T12:43:23.4194930Z In file included from ../../../extensions/jni/ExecuteJavaControllerService.cpp:19: 2023-04-14T12:43:23.4223740Z In file included from ../../../extensions/jni/ExecuteJavaControllerService.h:36: 2023-04-14T12:43:23.4255880Z In file included from ../../../extensions/jni/ClassRegistrar.h:37: 2023-04-14T12:43:23.4318350Z ../../../extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function for call to 'min' 2023-04-14T12:43:23.4380880Z int actual = static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, std::min(remaining, buffer_.size(); 2023-04-14T12:43:23.4422190Z ^~~~ 2023-04-14T12:43:23.4474240Z /Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:39:1: note: candidate template ignored: deduced conflicting types for parameter '_Tp' ('unsigned long long' vs. 'unsigned long') 2023-04-14T12:43:23.4497750Z min(const _Tp& __a, const _Tp& __b) 2023-04-14T12:43:23.4546180Z ^ 2023-04-14T12:43:23.4648060Z /Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:50:1: note: candidate template ignored: could not match 'initializer_list' against 'unsigned long long' 2023-04-14T12:43:23.4652360Z cd /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-build/google/cloud/storage && ccache /usr/local/opt/ccache/bin/ccache /usr/local/opt/ccache/libexec/c++ -DCURL_STATICLIB -DHAS_EXECINFO=1 -DUSE_REAL_ODBC_TEST_DRIVER -D_SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING -D_SILENCE_CXX20_REL_OPS_DEPRECATION_WARNING -I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-src -I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/absl-src -I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/crc32c-src/include -isystem /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-build -isystem /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/nlohmann -isystem /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/curl-install/include -isystem /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/zlib-install/include -isystem /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/libressl-install/include -I/usr/local/opt/flex/include -std=c++20 -DENABLE_JNI -DOPENSSL_SUPPORT -DSODIUM_STATIC=1 -Wall -Wextra -Werror -O3 -DNDEBUG -std=gnu++20 -isysroot /Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -fPIC -Wall -Wextra -Wconversion -Wno-sign-conversion -MD -MT _deps/google-cloud-cpp-build/google/cloud/storage/CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o -MF CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o.d -o CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o -c /Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-src/google/cloud/storage/internal/compute_engine_util.cc 2023-04-14T12:43:23.4671030Z min(initializer_list<_Tp> __t, _Compare __comp) 2023-04-14T12:43:23.4771990Z ^ 2023-04-14T12:43:23.4857020Z /Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:59:1: note: candidate function template not viable: requires single argument '__t', but 2 arguments were provided 2023-04-14T12:43:23.4958490Z min(initializer_list<_Tp> __t) 2023-04-14T12:43:23.4995100Z ^ 2023-04-14T12:43:23.5096600Z /Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:30:1: note: candidate function template not viable: requires 3 arguments, but 2 were provided 2023-04-14T12:43:23.5197380Z min(const _Tp& __a, const _Tp& __b, _Compare __comp) 2023-04-14T12:43:23.5298390Z ^ 2023-04-14T12:43:23.5351750Z 1 error generated. 2023-04-14T12:43:23.5453170Z make[2]: *** [extensions/jni/CMakeFiles/minifi-jni.dir/ExecuteJavaControllerService.cpp.o] Error 1 2023-04-14T12:43:23.350Z make[1]: *** [extensions/jni/CMakeFiles/minifi-jni.dir/all] Error 2 2023-04-14T12:43:23.5656530Z make[1]: *** Waiting for unfinished jobs -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2102) Compile failure with -ENABLE_LUA_SCRIPTING on aarch64
[ https://issues.apache.org/jira/browse/MINIFICPP-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-2102: --- Epic Link: MINIFICPP-2048 > Compile failure with -ENABLE_LUA_SCRIPTING on aarch64 > - > > Key: MINIFICPP-2102 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2102 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > > Due to missing luaopen_bit32 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-2102) Compile failure with -ENABLE_LUA_SCRIPTING on aarch64
Martin Zink created MINIFICPP-2102: -- Summary: Compile failure with -ENABLE_LUA_SCRIPTING on aarch64 Key: MINIFICPP-2102 URL: https://issues.apache.org/jira/browse/MINIFICPP-2102 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Reporter: Martin Zink Assignee: Martin Zink Due to missing luaopen_bit32 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2063) RFC3339 parsing with expression language
[ https://issues.apache.org/jira/browse/MINIFICPP-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-2063: Resolution: Done Status: Resolved (was: Patch Available) > RFC3339 parsing with expression language > > > Key: MINIFICPP-2063 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2063 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > RFC3339 complient strings can be tricky to parse with expression language, > because it accepts a lot of formats and toDate can only handle one specific > format specifier. > It would be nice if we had an expression language function that handles all > compliant string representations so we can parse them regardless of source. > ListenSyslog parsed messages are one example where the timestamp comes from > thirdparty applications and is semi regulated (it uses the subset of RFC3339). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-2030) Expose InFlightMessageCounter in PublishMQTT as a processor metric
[ https://issues.apache.org/jira/browse/MINIFICPP-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-2030. - Resolution: Done > Expose InFlightMessageCounter in PublishMQTT as a processor metric > -- > > Key: MINIFICPP-2030 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2030 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Ferenc Gerlits >Assignee: Ferenc Gerlits >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > (from > [https://github.com/apache/nifi-minifi-cpp/pull/1432#discussion_r1043352711] > by [~lordgamez]) > We can define processor specific metrics additionally to the general > processor metrics we have for all processors like transferred flow file > count, onTrigger calls etc. For example GetFileMetrics extends these metrics > with 2 additional ones. > `in_flight_message_count_` could be added as such an additional metric for > PublishMQTTMetrics -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-2095) Inconsistent naming in C2 DeviceInfoNode::systemInfo::machinearch
[ https://issues.apache.org/jira/browse/MINIFICPP-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-2095. - Resolution: Fixed > Inconsistent naming in C2 DeviceInfoNode::systemInfo::machinearch > - > > Key: MINIFICPP-2095 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2095 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Martin Zink >Assignee: Martin Zink >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Nifi and minifi java uses machineArch (the other properties also use > camelCase), whilst minifi cpp uses machinearch (this makes it hard to create > C2 servers that can be used simultaneously with java and cpp agents. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-2039) Dust off minificontroller
[ https://issues.apache.org/jira/browse/MINIFICPP-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-2039. - Resolution: Done > Dust off minificontroller > - > > Key: MINIFICPP-2039 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2039 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Gábor Gyimesi >Assignee: Gábor Gyimesi >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > Make sure all functionalities work properly, add test coverage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1528: MINIFICPP-2063 RFC3339 parsing with expression language
szaszm closed pull request #1528: MINIFICPP-2063 RFC3339 parsing with expression language URL: https://github.com/apache/nifi-minifi-cpp/pull/1528 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1527: MINIFICPP-2030 Expose InFlightMessageCounter in PublishMQTT as processor metric
szaszm closed pull request #1527: MINIFICPP-2030 Expose InFlightMessageCounter in PublishMQTT as processor metric URL: https://github.com/apache/nifi-minifi-cpp/pull/1527 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1503: MINIFICPP-2039 Dust off minificontroller
szaszm closed pull request #1503: MINIFICPP-2039 Dust off minificontroller URL: https://github.com/apache/nifi-minifi-cpp/pull/1503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1556: MINIFICPP-2095 fix inconsistent naming in C2 machineArch
szaszm closed pull request #1556: MINIFICPP-2095 fix inconsistent naming in C2 machineArch URL: https://github.com/apache/nifi-minifi-cpp/pull/1556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-11433) Upgrade moment to 2.29.4
[ https://issues.apache.org/jira/browse/NIFI-11433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan reassigned NIFI-11433: -- Assignee: Scott Aslan > Upgrade moment to 2.29.4 > > > Key: NIFI-11433 > URL: https://issues.apache.org/jira/browse/NIFI-11433 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.21.0 >Reporter: Mike R >Assignee: Scott Aslan >Priority: Major > > Upgrade moment to 2.29.4 to remediate vulnerability [Regular Expression > Denial of Service (ReDoS) in moment | CVE-2022-31129 | > Snyk|https://security.snyk.io/vuln/SNYK-JS-MOMENT-2944238] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] mark-bathori commented on pull request #7170: NIFI-11440: Speed up Hive Metastore based unit tests
mark-bathori commented on PR #7170: URL: https://github.com/apache/nifi/pull/7170#issuecomment-1508531983 Thanks @exceptionfactory for the review. I overlooked that the modified `AfterAll` would be still called if every test case is ignored on method level. I moved the `DisabledOnOs` annotation to class level. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] greyp9 commented on a diff in pull request #7173: NIFI-11439 - correct provenance reporting parameter
greyp9 commented on code in PR #7173: URL: https://github.com/apache/nifi/pull/7173#discussion_r1166843465 ## nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/FetchGCSObject.java: ## @@ -273,7 +275,15 @@ public void onTrigger(final ProcessContext context, final ProcessSession session final long millis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos); getLogger().info("Successfully retrieved GCS Object for {} in {} millis; routing to success", new Object[]{flowFile, millis}); -session.getProvenanceReporter().fetch(flowFile, "https://"; + bucketName + ".storage.googleapis.com/" + key, millis); + +String transitUri; +try { +final URL url = new URL(storage.getOptions().getHost()); +transitUri = String.format("%s://%s.%s/%s", url.getProtocol(), bucketName, url.getHost(), key); +} catch (MalformedURLException e) { +transitUri = e.getClass().getSimpleName(); +} Review Comment: Yep, that's better. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11449) add autocommit property to PutDatabaseRecord processor
[ https://issues.apache.org/jira/browse/NIFI-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdelrahim Ahmad updated NIFI-11449: Description: The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When using the processor with the Trino-JDBC-Driver or Dremio-JDBC-Driver to write to an Iceberg catalog, it disables the autocommit feature. This leads to errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}". the autocommit feature needs to be added in the processor to be enabled/disabled. enabling auto-commit in the Nifi PutDatabaseRecord processor is important for Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by allowing atomic writes to be performed in the underlying database. This will allow the process to be widely used with bigger range of databases. _Improving this processor will allow Nifi to be the main tool to ingest data into these new Technologies. So we don't have to deal with another tool to do so._ +*_{color:#de350b}BUT:{color}_*+ I have reviewed The {{PutDatabaseRecord}} processor in NiFi. It inserts records one by one into the database using a prepared statement, and commits the transaction at the end of the loop that processes each record. This approach can be inefficient and slow when inserting large volumes of data into tables that are optimized for bulk ingestion, such as Delta Lake, Iceberg, and Hudi tables. These tables use various techniques to optimize the performance of bulk ingestion, such as partitioning, clustering, and indexing. Inserting records one by one using a prepared statement can bypass these optimizations, leading to poor performance and potentially causing issues such as excessive disk usage, increased memory consumption, and decreased query performance. To avoid these issues, it is recommended to have a new processor, or add feature to the current one, to bulk insert method with AutoCommit feature when inserting large volumes of data into Delta Lake, Iceberg, and Hudi tables. P.S.: using PutSQL is not a have autoCommit but have the same performance problem described above.. Thanks and best regards :) Abdelrahim Ahmad was: The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When using the processor with the Trino-JDBC-Driver or Dremio-JDBC-Driver to write to an Iceberg catalog, it disables the autocommit feature. This leads to errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}". To fix this issue, the autocommit feature needs to be added in the processor to be enabled/disabled. enabling auto-commit in the Nifi PutDatabaseRecord processor is important for Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by allowing atomic writes to be performed in the underlying database. This will allow the process to be widely used with bigger range of databases. _*Improving this processor will allow Nifi to be the main tool to ingest data into these new Technologies. So we don't have to deal with another tool to do so.*_ P.S.: using PutSQL is not a good option at all due to the sensitivity of these tables when dealing with small inserts. Thanks and best regards Abdelrahim Ahmad > add autocommit property to PutDatabaseRecord processor > -- > > Key: NIFI-11449 > URL: https://issues.apache.org/jira/browse/NIFI-11449 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.21.0 > Environment: Any Nifi Deployment >Reporter: Abdelrahim Ahmad >Priority: Blocker > Labels: Trino, autocommit, database, iceberg, putdatabaserecord > > The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When > using the processor with the Trino-JDBC-Driver or Dremio-JDBC-Driver to write > to an Iceberg catalog, it disables the autocommit feature. This leads to > errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}". > the autocommit feature needs to be added in the processor to be > enabled/disabled. > enabling auto-commit in the Nifi PutDatabaseRecord processor is important for > Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by > allowing atomic writes to be performed in the underlying database. This will > allow the process to be widely used with bigger range of databases. > _Improving this processor will allow Nifi to be the main tool to ingest data > into these new Technologies. So we don't have to deal with another tool to do > so._ > +*_{color:#de350b}BUT:{color}_*+ > I have reviewed The {{PutDatabaseRecord}} processor in NiFi. It inserts > records one by one into the database using a prepared statement, and commits > the transaction at the end of the loop that processe
[jira] [Updated] (NIFI-11449) add autocommit property to PutDatabaseRecord processor
[ https://issues.apache.org/jira/browse/NIFI-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdelrahim Ahmad updated NIFI-11449: Issue Type: New Feature (was: Improvement) > add autocommit property to PutDatabaseRecord processor > -- > > Key: NIFI-11449 > URL: https://issues.apache.org/jira/browse/NIFI-11449 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.21.0 > Environment: Any Nifi Deployment >Reporter: Abdelrahim Ahmad >Priority: Blocker > Labels: Trino, autocommit, database, iceberg, putdatabaserecord > > The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When > using the processor with the Trino-JDBC-Driver or Dremio-JDBC-Driver to write > to an Iceberg catalog, it disables the autocommit feature. This leads to > errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}". > To fix this issue, the autocommit feature needs to be added in the processor > to be enabled/disabled. > enabling auto-commit in the Nifi PutDatabaseRecord processor is important for > Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by > allowing atomic writes to be performed in the underlying database. This will > allow the process to be widely used with bigger range of databases. > _*Improving this processor will allow Nifi to be the main tool to ingest data > into these new Technologies. So we don't have to deal with another tool to do > so.*_ > P.S.: using PutSQL is not a good option at all due to the sensitivity of > these tables when dealing with small inserts. > Thanks and best regards > Abdelrahim Ahmad -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-2101) Compilation failure in PutSFTPTests on MacOS
Martin Zink created MINIFICPP-2101: -- Summary: Compilation failure in PutSFTPTests on MacOS Key: MINIFICPP-2101 URL: https://issues.apache.org/jira/browse/MINIFICPP-2101 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Reporter: Martin Zink Assignee: Martin Zink file_clock::from_sys(modification_time) used in PutSFTPTests is not available on every supported platform -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-1184) Cannot compile on Raspberry Pi (Raspbian bullseye)
[ https://issues.apache.org/jira/browse/MINIFICPP-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink resolved MINIFICPP-1184. Resolution: Fixed this commit fixed all ARM compile issues [https://github.com/apache/nifi-minifi-cpp/commit/06d5467c6ca2d9a20ca9a969fd768e03025c362d] I was able to compile minifi on raspberry 3B+ (64 bit OS) > Cannot compile on Raspberry Pi (Raspbian bullseye) > -- > > Key: MINIFICPP-1184 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1184 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Affects Versions: 0.7.0 > Environment: Raspberry Pi 4 >Reporter: dan rey >Assignee: Dániel Bakai >Priority: Major > > I am unable to compile this on a Raspberry Pi running Raspbian GNU/Linux > bullseye. > I continually get an error about linking. > [ 59%] Linking CXX executable minifi > /usr/bin/ld: ../libminifi/libcore-minifi.a(Connection.cpp.o): in function > `std::__atomic_base::load(std::memory_order) const': > /usr/include/c++/9/bits/atomic_base.h:419: undefined reference to > `__atomic_load_8' > /usr/bin/ld: /usr/include/c++/9/bits/atomic_base.h:419: undefined reference > to `__atomic_load_8' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request, #1558: MINIFICPP-2100 Add RockyLinux Dockerfile
lordgamez opened a new pull request, #1558: URL: https://github.com/apache/nifi-minifi-cpp/pull/1558 https://issues.apache.org/jira/browse/MINIFICPP-2100 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-2100) Create Dockerfile for RockyLinux build
Gábor Gyimesi created MINIFICPP-2100: Summary: Create Dockerfile for RockyLinux build Key: MINIFICPP-2100 URL: https://issues.apache.org/jira/browse/MINIFICPP-2100 Project: Apache NiFi MiNiFi C++ Issue Type: New Feature Reporter: Gábor Gyimesi Assignee: Gábor Gyimesi Due to a bug described in https://issues.apache.org/jira/browse/MINIFICPP-2077 we cannot release convenience binaries for arm64 using the centos build. An alternative would be to use RockyLinux8 for this purpose, but for that we need to create a new Dockerfile. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] timeabarna commented on pull request #7092: NIFI-11327 Add Export/Import All - NiFi CLI - NiFi Registry
timeabarna commented on PR #7092: URL: https://github.com/apache/nifi/pull/7092#issuecomment-1508199943 Hello @bbende and @exceptionfactory, Thanks you very much for your review. I've updated the code based on your recommendations, improved serialisation and added storage location update. Can you please check again? Thanks in advance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org