[jira] [Updated] (NIFI-11241) Introduce API for building Processors in Python

2023-04-14 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-11241:

Fix Version/s: 2.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Introduce API for building Processors in Python
> ---
>
> Key: NIFI-11241
> URL: https://issues.apache.org/jira/browse/NIFI-11241
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework, Documentation & Website, Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> Introduce an API for building Processors in Python. Use Py4J for 
> communication layer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-11241) Introduce API for building Processors in Python

2023-04-14 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-11241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712518#comment-17712518
 ] 

ASF subversion and git services commented on NIFI-11241:


Commit 7f7cccfdc2f0788f5fc26703ac8e836092d6d06b in nifi's branch 
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7f7cccfdc2 ]

NIFI-11241 Initial implementation of Python-based Processor API with Py4J

- Updated GitHub workflow so that system tests include Python 3.9
- Updated GitHub actions to build necessary modules for system tests

This closes #7003

Co-authored-by: David Handermann 
Signed-off-by: David Handermann 


> Introduce API for building Processors in Python
> ---
>
> Key: NIFI-11241
> URL: https://issues.apache.org/jira/browse/NIFI-11241
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework, Documentation & Website, Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> Introduce an API for building Processors in Python. Use Py4J for 
> communication layer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[GitHub] [nifi] exceptionfactory closed pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…

2023-04-14 Thread via GitHub


exceptionfactory closed pull request #7003: NIFI-11241: Initial implementation 
of Python-based Processor API with…
URL: https://github.com/apache/nifi/pull/7003


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [nifi] mr1716 closed pull request #7162: NIFI-11432 Upgrade Angular Version to 1.8.3

2023-04-14 Thread via GitHub


mr1716 closed pull request #7162: NIFI-11432 Upgrade Angular Version to 1.8.3
URL: https://github.com/apache/nifi/pull/7162


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [nifi] scottyaslan commented on a diff in pull request #7162: NIFI-11432 Upgrade Angular Version to 1.8.3

2023-04-14 Thread via GitHub


scottyaslan commented on code in PR #7162:
URL: https://github.com/apache/nifi/pull/7162#discussion_r1167154119


##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/frontend/package.json:
##
@@ -20,7 +20,7 @@
 "preinstall": "npm --ignore-scripts install npm-force-resolutions && 
npm-force-resolutions"
   },
   "dependencies": {
-"angular": "1.8.2",
+"angular": "1.8.3",
 "angular-animate": "1.8.2",

Review Comment:
   FYI, these other angular-* packages should also be updated when upgrading 
angular.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [nifi] scottyaslan commented on a diff in pull request #7162: NIFI-11432 Upgrade Angular Version to 1.8.3

2023-04-14 Thread via GitHub


scottyaslan commented on code in PR #7162:
URL: https://github.com/apache/nifi/pull/7162#discussion_r1163239177


##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/frontend/package-lock.json:
##
@@ -7,7 +7,7 @@
   "name": "apache-nifi",
   "hasInstallScript": true,
   "dependencies": {
-"angular": "1.8.2",
+"angular": "1.8.3",

Review Comment:
   This is not correct. Anytime the dependencies are changed in the 
package.json the package-lock.json should be deleted and regenerated with the 
appropriate version of npm (v8.1.2). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Resolved] (NIFI-11440) Speed up Hive Metastore-based tests for Hive 3 and Iceberg

2023-04-14 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann resolved NIFI-11440.
-
Fix Version/s: 2.0.0
   1.22.0
   Resolution: Fixed

> Speed up Hive Metastore-based tests for Hive 3 and Iceberg
> --
>
> Key: NIFI-11440
> URL: https://issues.apache.org/jira/browse/NIFI-11440
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mark Bathori
>Assignee: Mark Bathori
>Priority: Major
> Fix For: 2.0.0, 1.22.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The Hive Metastore based unit tests take a lot of time because the metastore 
> is initialized before every test. These executions can be speed up by 
> changing the initialization to be called only once before starting the tests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-11382) UI - Processor property configurations throws console error when referenced parameter is deleted

2023-04-14 Thread Scott Aslan (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-11382:
---
Description: 
When a processor is configured with a referenced parameter and that parameter 
is deleted, the next time the user configures the property, it will throw a 
console error.

 

This is related to NIFI-11451

  was:When a processor is configured with a referenced parameter and that 
parameter is deleted, the next time the user configures the property, it will 
throw a console error.


> UI - Processor property configurations throws console error when referenced 
> parameter is deleted
> 
>
> Key: NIFI-11382
> URL: https://issues.apache.org/jira/browse/NIFI-11382
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.20.0
>Reporter: Margot Tien
>Assignee: Margot Tien
>Priority: Minor
> Attachments: Screen Shot 2023-03-31 at 12.36.32 PM.png
>
>
> When a processor is configured with a referenced parameter and that parameter 
> is deleted, the next time the user configures the property, it will throw a 
> console error.
>  
> This is related to NIFI-11451



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-11451) UI - property table console error

2023-04-14 Thread Scott Aslan (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-11451:
---
Description: 
When attempting to reference a parameter or create a controller service inline 
there is a console error in devTools:

 

This is related to NIFI-11382

  was:
When attempting to reference a parameter or create a controller service inline 
there is a console error in devTools:

 

 


> UI - property table console error
> -
>
> Key: NIFI-11451
> URL: https://issues.apache.org/jira/browse/NIFI-11451
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Scott Aslan
>Priority: Major
> Attachments: Screenshot 2023-04-14 at 1.10.58 PM.png
>
>
> When attempting to reference a parameter or create a controller service 
> inline there is a console error in devTools:
>  
> This is related to NIFI-11382



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-11440) Speed up Hive Metastore-based tests for Hive 3 and Iceberg

2023-04-14 Thread David Handermann (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-11440:

Summary: Speed up Hive Metastore-based tests for Hive 3 and Iceberg  (was: 
Speed up Hive Metastore based unit tests)

> Speed up Hive Metastore-based tests for Hive 3 and Iceberg
> --
>
> Key: NIFI-11440
> URL: https://issues.apache.org/jira/browse/NIFI-11440
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mark Bathori
>Assignee: Mark Bathori
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The Hive Metastore based unit tests take a lot of time because the metastore 
> is initialized before every test. These executions can be speed up by 
> changing the initialization to be called only once before starting the tests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-11440) Speed up Hive Metastore based unit tests

2023-04-14 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712488#comment-17712488
 ] 

ASF subversion and git services commented on NIFI-11440:


Commit 15a1134c85c38e1d6fe146259ca8b2f77c6cdbf5 in nifi's branch 
refs/heads/support/nifi-1.x from Mark Bathori
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=15a1134c85 ]

NIFI-11440 Speed up Iceberg Hive Metastore Tests

This closes #7170

Signed-off-by: David Handermann 
(cherry picked from commit 44a7f7f38b3950a8a0135d1ce8602a4890c3d995)


> Speed up Hive Metastore based unit tests
> 
>
> Key: NIFI-11440
> URL: https://issues.apache.org/jira/browse/NIFI-11440
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mark Bathori
>Assignee: Mark Bathori
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The Hive Metastore based unit tests take a lot of time because the metastore 
> is initialized before every test. These executions can be speed up by 
> changing the initialization to be called only once before starting the tests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-11440) Speed up Hive Metastore based unit tests

2023-04-14 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712487#comment-17712487
 ] 

ASF subversion and git services commented on NIFI-11440:


Commit 44a7f7f38b3950a8a0135d1ce8602a4890c3d995 in nifi's branch 
refs/heads/main from Mark Bathori
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=44a7f7f38b ]

NIFI-11440 Speed up Iceberg Hive Metastore Tests

This closes #7170

Signed-off-by: David Handermann 


> Speed up Hive Metastore based unit tests
> 
>
> Key: NIFI-11440
> URL: https://issues.apache.org/jira/browse/NIFI-11440
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mark Bathori
>Assignee: Mark Bathori
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The Hive Metastore based unit tests take a lot of time because the metastore 
> is initialized before every test. These executions can be speed up by 
> changing the initialization to be called only once before starting the tests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[GitHub] [nifi] exceptionfactory closed pull request #7170: NIFI-11440: Speed up Hive Metastore based unit tests

2023-04-14 Thread via GitHub


exceptionfactory closed pull request #7170: NIFI-11440: Speed up Hive Metastore 
based unit tests
URL: https://github.com/apache/nifi/pull/7170


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [nifi] scottyaslan opened a new pull request, #7174: [NIFIF-11433] update angular, d3, moment, slickgrid, and jquery depen…

2023-04-14 Thread via GitHub


scottyaslan opened a new pull request, #7174:
URL: https://github.com/apache/nifi/pull/7174

   …dencies
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   # Summary
   
   [NIFI-11433](https://issues.apache.org/jira/browse/NIFI-11433)
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [X] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [X] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [X] Pull Request based on current revision of the `main` branch
   - [X] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [X] Build completed using `mvn clean install -P contrib-check`
 - [X] JDK 11
 - [ ] JDK 17
   
   ### Licensing
   
   - [X] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [X] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [X] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (NIFI-11451) UI - property table console error

2023-04-14 Thread Scott Aslan (Jira)
Scott Aslan created NIFI-11451:
--

 Summary: UI - property table console error
 Key: NIFI-11451
 URL: https://issues.apache.org/jira/browse/NIFI-11451
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Scott Aslan
 Attachments: Screenshot 2023-04-14 at 1.10.58 PM.png

When attempting to reference a parameter or create a controller service inline 
there is a console error in devTools:

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-11433) Upgrade moment to 2.29.4

2023-04-14 Thread Scott Aslan (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-11433:
---
Description: 
Upgrade d3 to 7.6.1 to be able to remove d3-selection-multi.

Upgrade Angular and related packages to 1.8.3

Upgrade to jQuery 3.6.4

Upgrade to slickgrid 2.4.45

Upgrade moment to 2.29.4 to remediate vulnerability [Regular Expression Denial 
of Service (ReDoS) in moment | CVE-2022-31129 | 
Snyk|https://security.snyk.io/vuln/SNYK-JS-MOMENT-2944238]

  was:Upgrade moment to 2.29.4 to remediate vulnerability [Regular Expression 
Denial of Service (ReDoS) in moment | CVE-2022-31129 | 
Snyk|https://security.snyk.io/vuln/SNYK-JS-MOMENT-2944238]


> Upgrade moment to 2.29.4
> 
>
> Key: NIFI-11433
> URL: https://issues.apache.org/jira/browse/NIFI-11433
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.21.0
>Reporter: Mike R
>Assignee: Scott Aslan
>Priority: Major
>
> Upgrade d3 to 7.6.1 to be able to remove d3-selection-multi.
> Upgrade Angular and related packages to 1.8.3
> Upgrade to jQuery 3.6.4
> Upgrade to slickgrid 2.4.45
> Upgrade moment to 2.29.4 to remediate vulnerability [Regular Expression 
> Denial of Service (ReDoS) in moment | CVE-2022-31129 | 
> Snyk|https://security.snyk.io/vuln/SNYK-JS-MOMENT-2944238]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (MINIFICPP-2104) CWEL Output Format validation

2023-04-14 Thread Marton Szasz (Jira)
Marton Szasz created MINIFICPP-2104:
---

 Summary: CWEL Output Format validation
 Key: MINIFICPP-2104
 URL: https://issues.apache.org/jira/browse/MINIFICPP-2104
 Project: Apache NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: Marton Szasz


When using an invalid output format, like 'Text' instead of 'Plaintext', CWEL 
prints no errors, runs normally, but doesn't output anything. We should improve 
property validation to give an error message, instead of silently failing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (NIFI-11450) Node respecting nifi.flowcontroller.autoResumeState when joining cluster

2023-04-14 Thread Phil Lord (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phil Lord updated NIFI-11450:
-
Environment: Nifi version 1.19.1.  Linux.  JDK11+  (was: Nifi version 11.  
Linux.  JDK11+)

> Node respecting nifi.flowcontroller.autoResumeState when joining cluster
> 
>
> Key: NIFI-11450
> URL: https://issues.apache.org/jira/browse/NIFI-11450
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.19.1
> Environment: Nifi version 1.19.1.  Linux.  JDK11+
>Reporter: Phil Lord
>Priority: Critical
>
> Mistakenly tried to connect a node to an active cluster that had the 
> following setting: nifi.flowcontroller.autoResumeState=true .
> The flow.xml/json were both removed completely prior to attempting to connect 
> node to the cluster, then started the node.  The node connected to the 
> cluster without issue, however all components/controllerServices on that node 
> were in a stopped state(due to the property mentioned above).
> Is this correct behavior given that the flow that was inherited from the 
> cluster had components started/etc.  I would think the node should either 
> fail to join the cluster/start?
> This was running Nifi 1.19.1... unclear if it's resolved in latest release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (NIFI-11450) Node respecting nifi.flowcontroller.autoResumeState when joining cluster

2023-04-14 Thread Phil Lord (Jira)
Phil Lord created NIFI-11450:


 Summary: Node respecting nifi.flowcontroller.autoResumeState when 
joining cluster
 Key: NIFI-11450
 URL: https://issues.apache.org/jira/browse/NIFI-11450
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.19.1
 Environment: Nifi version 11.  Linux.  JDK11+
Reporter: Phil Lord


Mistakenly tried to connect a node to an active cluster that had the following 
setting: nifi.flowcontroller.autoResumeState=true .

The flow.xml/json were both removed completely prior to attempting to connect 
node to the cluster, then started the node.  The node connected to the cluster 
without issue, however all components/controllerServices on that node were in a 
stopped state(due to the property mentioned above).

Is this correct behavior given that the flow that was inherited from the 
cluster had components started/etc.  I would think the node should either fail 
to join the cluster/start?

This was running Nifi 1.19.1... unclear if it's resolved in latest release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…

2023-04-14 Thread via GitHub


exceptionfactory commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1167047760


##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.py4j;
+
+import org.apache.nifi.py4j.client.JavaObjectBindings;
+import org.apache.nifi.py4j.client.NiFiPythonGateway;
+import org.apache.nifi.py4j.client.StandardPythonClient;
+import org.apache.nifi.py4j.server.NiFiGatewayServer;
+import org.apache.nifi.python.ControllerServiceTypeLookup;
+import org.apache.nifi.python.PythonController;
+import org.apache.nifi.python.PythonProcessConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import py4j.CallbackClient;
+import py4j.GatewayServer;
+
+import javax.net.ServerSocketFactory;
+import javax.net.SocketFactory;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+
+// TODO / Figure Out for MVP:
+//  MUST DO:
+//  - Documentation
+//  - Admin Guide
+//  - JavaDocs
+//  - Developer Guide
+//  - Explain how communication between Java & Python work.
+//  - Java is preferred, Python is slower and more expensive b/c 
of network
+//  - Different Extension Points (FlowFileTransform, 
RecordTransform)
+//  - What the API Looks like, Links to JavaDocs for 
ProcessContext, etc.
+//  - Example Code
+//  - Exposing properties
+//  - Relationships
+//  - Controller Services
+//  - Need to update docs to show the interfaces that are 
exposed, explain how to get these...
+//  - Design Doc
+//  - Setup proper logging on the Python side: 
https://docs.python.org/2/howto/logging-cookbook.html#using-file-rotation
+//  - For FlowFileTransform, allow the result to contain either a byte 
array or a String. If a String, just convert in the parent class.
+//  - Figure out how to deal with Python Packaging
+//  - Need to figure out how to deal with additionalDetails.html, 
docs directory in python project typically?
+//  - Understand how to deal with versioning
+//  - Look at performance improvements for Py4J - socket comms appear to 
be INCREDIBLY slow.
+//  - Create test that calls Python 1M times. Just returns 
'hello'. See how long it takes
+//  - Create test that calls Python 1M times. Returns .toString() and see how long it takes.
+//  - Will help to understand if it's the call from Java to Python 
that's slow, Python to Java, or both.
+//  - Performance concern for TransformRecord
+//  - Currently, triggering the transform() method is pretty fast. 
But then the Result object comes back and we have to call into the Python side 
to call the getters
+//over and over. Need to look into instead serializing the 
entire response as JSON and sending that back.
+//  - Also, since this is heavy JSON processing, might want to 
consider ORJSON or something like that instead of inbuilt JSON parser/generator
+//  - Test pip install nifi-my-proc, does nifi pick it up?
+//  - When ran DetectObjectInImage with multiple threads, Python died. 
Need to figure out why.
+//  - If Python Process dies, need to create a new process and need to 
then create all of the Processors that were in that Process and initialize them.
+//- Milestone 2 or 3, not Milestone 1.
+//  - Remove test-pypi usage from ExtensionManager.py
+//  - Additional Interfaces beyond just FlowFileTransform
+//  - FlowFileSource
+//  - Restructure Maven projects
+//  - Should this all go under Framework?
+//
+//
+//  CONSIDER:
+//  - Clustering: Ensure component on all nodes?
+//  - Consider "pip freeze" type of thing to ensure that python 
dependencies are same across nodes when joining cluster.
+//  - Update pytho

[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…

2023-04-14 Thread via GitHub


exceptionfactory commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1167045800


##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.py4j;
+
+import org.apache.nifi.py4j.client.JavaObjectBindings;
+import org.apache.nifi.py4j.client.NiFiPythonGateway;
+import org.apache.nifi.py4j.client.StandardPythonClient;
+import org.apache.nifi.py4j.server.NiFiGatewayServer;
+import org.apache.nifi.python.ControllerServiceTypeLookup;
+import org.apache.nifi.python.PythonController;
+import org.apache.nifi.python.PythonProcessConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import py4j.CallbackClient;
+import py4j.GatewayServer;
+
+import javax.net.ServerSocketFactory;
+import javax.net.SocketFactory;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+
+// TODO / Figure Out for MVP:
+//  MUST DO:
+//  - Documentation
+//  - Admin Guide
+//  - JavaDocs
+//  - Developer Guide
+//  - Explain how communication between Java & Python work.
+//  - Java is preferred, Python is slower and more expensive b/c 
of network
+//  - Different Extension Points (FlowFileTransform, 
RecordTransform)
+//  - What the API Looks like, Links to JavaDocs for 
ProcessContext, etc.
+//  - Example Code
+//  - Exposing properties
+//  - Relationships
+//  - Controller Services
+//  - Need to update docs to show the interfaces that are 
exposed, explain how to get these...
+//  - Design Doc
+//  - Setup proper logging on the Python side: 
https://docs.python.org/2/howto/logging-cookbook.html#using-file-rotation
+//  - For FlowFileTransform, allow the result to contain either a byte 
array or a String. If a String, just convert in the parent class.
+//  - Figure out how to deal with Python Packaging
+//  - Need to figure out how to deal with additionalDetails.html, 
docs directory in python project typically?
+//  - Understand how to deal with versioning
+//  - Look at performance improvements for Py4J - socket comms appear to 
be INCREDIBLY slow.
+//  - Create test that calls Python 1M times. Just returns 
'hello'. See how long it takes
+//  - Create test that calls Python 1M times. Returns .toString() and see how long it takes.
+//  - Will help to understand if it's the call from Java to Python 
that's slow, Python to Java, or both.
+//  - Performance concern for TransformRecord
+//  - Currently, triggering the transform() method is pretty fast. 
But then the Result object comes back and we have to call into the Python side 
to call the getters
+//over and over. Need to look into instead serializing the 
entire response as JSON and sending that back.
+//  - Also, since this is heavy JSON processing, might want to 
consider ORJSON or something like that instead of inbuilt JSON parser/generator
+//  - Test pip install nifi-my-proc, does nifi pick it up?
+//  - When ran DetectObjectInImage with multiple threads, Python died. 
Need to figure out why.
+//  - If Python Process dies, need to create a new process and need to 
then create all of the Processors that were in that Process and initialize them.
+//- Milestone 2 or 3, not Milestone 1.
+//  - Remove test-pypi usage from ExtensionManager.py
+//  - Additional Interfaces beyond just FlowFileTransform
+//  - FlowFileSource
+//  - Restructure Maven projects
+//  - Should this all go under Framework?
+//
+//
+//  CONSIDER:
+//  - Clustering: Ensure component on all nodes?
+//  - Consider "pip freeze" type of thing to ensure that python 
dependencies are same across nodes when joining cluster.
+//  - Update pytho

[jira] [Updated] (MINIFICPP-2084) Reverse DNS timeout test fails transiently

2023-04-14 Thread Marton Szasz (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Szasz updated MINIFICPP-2084:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Reverse DNS timeout test fails transiently
> --
>
> Key: MINIFICPP-2084
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2084
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> It is due to the fact that the timeout uses a separate thread, and it can 
> happen that the reverse dns lookup finishes before the 0ms timeout even 
> starts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (MINIFICPP-2073) Separate docker build from docker tests in CI

2023-04-14 Thread Marton Szasz (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Szasz updated MINIFICPP-2073:

Resolution: Done
Status: Resolved  (was: Patch Available)

> Separate docker build from docker tests in CI
> -
>
> Key: MINIFICPP-2073
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2073
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Minor
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Since the docker tests are not as consistent as we would like, it would be 
> nice if we could rerun just the tests without the need to rebuild the image.
> We could use docker save/load image and github upload/download artifact to 
> achive this capability.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (MINIFICPP-2018) Add ProcessContext::getStateManager to Lua and Python API

2023-04-14 Thread Marton Szasz (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Szasz updated MINIFICPP-2018:

Resolution: Done
Status: Resolved  (was: Patch Available)

> Add ProcessContext::getStateManager to Lua and Python API
> -
>
> Key: MINIFICPP-2018
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2018
> Project: Apache NiFi MiNiFi C++
>  Issue Type: New Feature
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1538: MINIFICPP-2073 Separate docker build from docker tests in CI

2023-04-14 Thread via GitHub


szaszm closed pull request #1538: MINIFICPP-2073 Separate docker build from 
docker tests in CI
URL: https://github.com/apache/nifi-minifi-cpp/pull/1538


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1541: MINIFICPP-2084 Fix flaky Reverse DNS timeout test

2023-04-14 Thread via GitHub


szaszm closed pull request #1541: MINIFICPP-2084 Fix flaky Reverse DNS timeout 
test
URL: https://github.com/apache/nifi-minifi-cpp/pull/1541


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1530: MINIFICPP-2018 Add ProcessContext::getStateManager to Lua/Python

2023-04-14 Thread via GitHub


szaszm closed pull request #1530: MINIFICPP-2018 Add 
ProcessContext::getStateManager to Lua/Python
URL: https://github.com/apache/nifi-minifi-cpp/pull/1530


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (MINIFICPP-2103) JNI extension fails to compile with clang

2023-04-14 Thread Martin Zink (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Zink updated MINIFICPP-2103:
---
Summary: JNI extension fails to compile with clang  (was: JNI extension 
fails to compile on macOS)

> JNI extension fails to compile with clang
> -
>
> Key: MINIFICPP-2103
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2103
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
>
> extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function 
> for call to 'min'
>       int actual = 
> static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, 
> std::min(remaining, buffer_.size();
>                                                                               
>                                                                          
> ^~~~



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[GitHub] [nifi] exceptionfactory commented on pull request #7170: NIFI-11440: Speed up Hive Metastore based unit tests

2023-04-14 Thread via GitHub


exceptionfactory commented on PR #7170:
URL: https://github.com/apache/nifi/pull/7170#issuecomment-1508798033

   Thanks for making the adjustments @mark-bathori! I started the automated 
build and will plan on merging pending successful completion on all platforms.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Commented] (NIFI-11439) Add Storage API URL to GCS Processors

2023-04-14 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712439#comment-17712439
 ] 

ASF subversion and git services commented on NIFI-11439:


Commit d762f35345a8486903a0b30d0923e1e8a3377863 in nifi's branch 
refs/heads/support/nifi-1.x from Paul Grey
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d762f35345 ]

NIFI-11439 Corrected GCS Transit URI for custom Storage API URL

This closes #7173

Signed-off-by: David Handermann 
(cherry picked from commit cff7808543716da5ca9f93356d7d762e15fdd143)


> Add Storage API URL to GCS Processors
> -
>
> Key: NIFI-11439
> URL: https://issues.apache.org/jira/browse/NIFI-11439
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Paul Grey
>Assignee: Paul Grey
>Priority: Minor
> Fix For: 2.0.0, 1.22.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> AWS processors allow the override of the base endpoint for API calls to AWS 
> services [1].  GCS libraries provide analogous capabilities, as documented 
> here [2].  Expose a GCS processor property to enable GCS endpoint override 
> behavior.
> [1] 
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java#L146-L154
> [2] 
> https://cloud.google.com/storage/docs/request-endpoints#storage-set-client-endpoint-java



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (NIFI-11439) Add Storage API URL to GCS Processors

2023-04-14 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/NIFI-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712438#comment-17712438
 ] 

ASF subversion and git services commented on NIFI-11439:


Commit cff7808543716da5ca9f93356d7d762e15fdd143 in nifi's branch 
refs/heads/main from Paul Grey
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=cff7808543 ]

NIFI-11439 Corrected GCS Transit URI for custom Storage API URL

This closes #7173

Signed-off-by: David Handermann 


> Add Storage API URL to GCS Processors
> -
>
> Key: NIFI-11439
> URL: https://issues.apache.org/jira/browse/NIFI-11439
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Paul Grey
>Assignee: Paul Grey
>Priority: Minor
> Fix For: 2.0.0, 1.22.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> AWS processors allow the override of the base endpoint for API calls to AWS 
> services [1].  GCS libraries provide analogous capabilities, as documented 
> here [2].  Expose a GCS processor property to enable GCS endpoint override 
> behavior.
> [1] 
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java#L146-L154
> [2] 
> https://cloud.google.com/storage/docs/request-endpoints#storage-set-client-endpoint-java



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[GitHub] [nifi] exceptionfactory closed pull request #7173: NIFI-11439 - correct provenance reporting parameter

2023-04-14 Thread via GitHub


exceptionfactory closed pull request #7173: NIFI-11439 - correct provenance 
reporting parameter
URL: https://github.com/apache/nifi/pull/7173


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (MINIFICPP-2103) JNI extension fails to compile on macOS

2023-04-14 Thread Martin Zink (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Zink updated MINIFICPP-2103:
---
Description: 
extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function 
for call to 'min'
      int actual = 
static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, 
std::min(remaining, buffer_.size();
                                                                                
                                                                       ^~~~

  was:
extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function 
for call to 'min'
      int actual = 
static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, 
std::min(remaining, buffer_.size();


> JNI extension fails to compile on macOS
> ---
>
> Key: MINIFICPP-2103
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2103
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
>
> extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function 
> for call to 'min'
>       int actual = 
> static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, 
> std::min(remaining, buffer_.size();
>                                                                               
>                                                                          
> ^~~~



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (MINIFICPP-2103) JNI extension fails to compile on macOS

2023-04-14 Thread Martin Zink (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Zink updated MINIFICPP-2103:
---
Description: 
extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching function 
for call to 'min'
      int actual = 
static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, 
std::min(remaining, buffer_.size();

  was:2023-04-14T12:43:23.4194930Z In file included from 
../../../extensions/jni/ExecuteJavaControllerService.cpp:19: 
2023-04-14T12:43:23.4223740Z In file included from 
../../../extensions/jni/ExecuteJavaControllerService.h:36: 
2023-04-14T12:43:23.4255880Z In file included from 
../../../extensions/jni/ClassRegistrar.h:37: 2023-04-14T12:43:23.4318350Z 
../../../extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching 
function for call to 'min' 2023-04-14T12:43:23.4380880Z int actual = 
static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, 
std::min(remaining, buffer_.size(); 2023-04-14T12:43:23.4422190Z ^~~~ 
2023-04-14T12:43:23.4474240Z 
/Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:39:1:
 note: candidate template ignored: deduced conflicting types for parameter 
'_Tp' ('unsigned long long' vs. 'unsigned long') 2023-04-14T12:43:23.4497750Z 
min(const _Tp& __a, const _Tp& __b) 2023-04-14T12:43:23.4546180Z ^ 
2023-04-14T12:43:23.4648060Z 
/Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:50:1:
 note: candidate template ignored: could not match 
'initializer_list' against 'unsigned long long' 
2023-04-14T12:43:23.4652360Z cd 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-build/google/cloud/storage
 && ccache /usr/local/opt/ccache/bin/ccache /usr/local/opt/ccache/libexec/c++ 
-DCURL_STATICLIB -DHAS_EXECINFO=1 -DUSE_REAL_ODBC_TEST_DRIVER 
-D_SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING 
-D_SILENCE_CXX20_REL_OPS_DEPRECATION_WARNING 
-I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-src
 -I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/absl-src 
-I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/crc32c-src/include
 -isystem 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-build
 -isystem 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/nlohmann 
-isystem 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/curl-install/include
 -isystem 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/zlib-install/include
 -isystem 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/libressl-install/include
 -I/usr/local/opt/flex/include -std=c++20 -DENABLE_JNI -DOPENSSL_SUPPORT 
-DSODIUM_STATIC=1 -Wall -Wextra -Werror -O3 -DNDEBUG -std=gnu++20 -isysroot 
/Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk
 -fPIC -Wall -Wextra -Wconversion -Wno-sign-conversion -MD -MT 
_deps/google-cloud-cpp-build/google/cloud/storage/CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o
 -MF 
CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o.d -o 
CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o -c 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-src/google/cloud/storage/internal/compute_engine_util.cc
 2023-04-14T12:43:23.4671030Z min(initializer_list<_Tp> __t, _Compare __comp) 
2023-04-14T12:43:23.4771990Z ^ 2023-04-14T12:43:23.4857020Z 
/Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:59:1:
 note: candidate function template not viable: requires single argument '__t', 
but 2 arguments were provided 2023-04-14T12:43:23.4958490Z 
min(initializer_list<_Tp> __t) 2023-04-14T12:43:23.4995100Z ^ 
2023-04-14T12:43:23.5096600Z 
/Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:30:1:
 note: candidate function template not viable: requires 3 arguments, but 2 were 
provided 2023-04-14T12:43:23.5197380Z min(const _Tp& __a, const _Tp& __b, 
_Compare __comp) 2023-04-14T12:43:23.5298390Z ^ 2023-04-14T12:43:23.5351750Z 1 
error generated. 2023-04-14T12:43:23.5453170Z make[2]: *** 
[extensions/jni/CMakeFiles/minifi-jni.dir/ExecuteJavaControllerService.cpp.o] 
Error 1 2023-04-14T12:43:23.350Z make[1]: *** 
[extensions/jni/CMakeFiles/minifi-jni.dir/all] Error 2 
2023-04-14T12:43:23.5656530Z make[1]: *** Waiting for unfinished jobs


> JNI extension fails to compile on macOS
> ---
>
> Key: MINIFICPP-2103
>

[jira] [Created] (MINIFICPP-2103) JNI extension fails to compile on macOS

2023-04-14 Thread Martin Zink (Jira)
Martin Zink created MINIFICPP-2103:
--

 Summary: JNI extension fails to compile on macOS
 Key: MINIFICPP-2103
 URL: https://issues.apache.org/jira/browse/MINIFICPP-2103
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Reporter: Martin Zink
Assignee: Martin Zink


2023-04-14T12:43:23.4194930Z In file included from 
../../../extensions/jni/ExecuteJavaControllerService.cpp:19: 
2023-04-14T12:43:23.4223740Z In file included from 
../../../extensions/jni/ExecuteJavaControllerService.h:36: 
2023-04-14T12:43:23.4255880Z In file included from 
../../../extensions/jni/ClassRegistrar.h:37: 2023-04-14T12:43:23.4318350Z 
../../../extensions/jni/jvm/JniReferenceObjects.h:112:86: error: no matching 
function for call to 'min' 2023-04-14T12:43:23.4380880Z int actual = 
static_cast(stream_->read(gsl::make_span(buffer_).subspan(0, 
std::min(remaining, buffer_.size(); 2023-04-14T12:43:23.4422190Z ^~~~ 
2023-04-14T12:43:23.4474240Z 
/Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:39:1:
 note: candidate template ignored: deduced conflicting types for parameter 
'_Tp' ('unsigned long long' vs. 'unsigned long') 2023-04-14T12:43:23.4497750Z 
min(const _Tp& __a, const _Tp& __b) 2023-04-14T12:43:23.4546180Z ^ 
2023-04-14T12:43:23.4648060Z 
/Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:50:1:
 note: candidate template ignored: could not match 
'initializer_list' against 'unsigned long long' 
2023-04-14T12:43:23.4652360Z cd 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-build/google/cloud/storage
 && ccache /usr/local/opt/ccache/bin/ccache /usr/local/opt/ccache/libexec/c++ 
-DCURL_STATICLIB -DHAS_EXECINFO=1 -DUSE_REAL_ODBC_TEST_DRIVER 
-D_SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING 
-D_SILENCE_CXX20_REL_OPS_DEPRECATION_WARNING 
-I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-src
 -I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/absl-src 
-I/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/crc32c-src/include
 -isystem 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-build
 -isystem 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/nlohmann 
-isystem 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/curl-install/include
 -isystem 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/zlib-install/include
 -isystem 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/thirdparty/libressl-install/include
 -I/usr/local/opt/flex/include -std=c++20 -DENABLE_JNI -DOPENSSL_SUPPORT 
-DSODIUM_STATIC=1 -Wall -Wextra -Werror -O3 -DNDEBUG -std=gnu++20 -isysroot 
/Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk
 -fPIC -Wall -Wextra -Wconversion -Wno-sign-conversion -MD -MT 
_deps/google-cloud-cpp-build/google/cloud/storage/CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o
 -MF 
CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o.d -o 
CMakeFiles/google_cloud_cpp_storage.dir/internal/compute_engine_util.cc.o -c 
/Users/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/build/_deps/google-cloud-cpp-src/google/cloud/storage/internal/compute_engine_util.cc
 2023-04-14T12:43:23.4671030Z min(initializer_list<_Tp> __t, _Compare __comp) 
2023-04-14T12:43:23.4771990Z ^ 2023-04-14T12:43:23.4857020Z 
/Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:59:1:
 note: candidate function template not viable: requires single argument '__t', 
but 2 arguments were provided 2023-04-14T12:43:23.4958490Z 
min(initializer_list<_Tp> __t) 2023-04-14T12:43:23.4995100Z ^ 
2023-04-14T12:43:23.5096600Z 
/Applications/Xcode_14.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/c++/v1/__algorithm/min.h:30:1:
 note: candidate function template not viable: requires 3 arguments, but 2 were 
provided 2023-04-14T12:43:23.5197380Z min(const _Tp& __a, const _Tp& __b, 
_Compare __comp) 2023-04-14T12:43:23.5298390Z ^ 2023-04-14T12:43:23.5351750Z 1 
error generated. 2023-04-14T12:43:23.5453170Z make[2]: *** 
[extensions/jni/CMakeFiles/minifi-jni.dir/ExecuteJavaControllerService.cpp.o] 
Error 1 2023-04-14T12:43:23.350Z make[1]: *** 
[extensions/jni/CMakeFiles/minifi-jni.dir/all] Error 2 
2023-04-14T12:43:23.5656530Z make[1]: *** Waiting for unfinished jobs



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (MINIFICPP-2102) Compile failure with -ENABLE_LUA_SCRIPTING on aarch64

2023-04-14 Thread Martin Zink (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Zink updated MINIFICPP-2102:
---
Epic Link: MINIFICPP-2048

> Compile failure with -ENABLE_LUA_SCRIPTING on aarch64
> -
>
> Key: MINIFICPP-2102
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2102
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
>
> Due to missing luaopen_bit32



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (MINIFICPP-2102) Compile failure with -ENABLE_LUA_SCRIPTING on aarch64

2023-04-14 Thread Martin Zink (Jira)
Martin Zink created MINIFICPP-2102:
--

 Summary: Compile failure with -ENABLE_LUA_SCRIPTING on aarch64
 Key: MINIFICPP-2102
 URL: https://issues.apache.org/jira/browse/MINIFICPP-2102
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Reporter: Martin Zink
Assignee: Martin Zink


Due to missing luaopen_bit32



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (MINIFICPP-2063) RFC3339 parsing with expression language

2023-04-14 Thread Marton Szasz (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Szasz updated MINIFICPP-2063:

Resolution: Done
Status: Resolved  (was: Patch Available)

> RFC3339 parsing with expression language
> 
>
> Key: MINIFICPP-2063
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2063
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> RFC3339 complient strings can be tricky to parse with expression language, 
> because it accepts a lot of formats and toDate can only handle one specific 
> format specifier.
> It would be nice if we had an expression language function that handles all 
> compliant string representations so we can parse them regardless of source. 
> ListenSyslog parsed messages are one example where the timestamp comes from 
> thirdparty applications and is semi regulated (it uses the subset of RFC3339).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (MINIFICPP-2030) Expose InFlightMessageCounter in PublishMQTT as a processor metric

2023-04-14 Thread Marton Szasz (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Szasz resolved MINIFICPP-2030.
-
Resolution: Done

> Expose InFlightMessageCounter in PublishMQTT as a processor metric
> --
>
> Key: MINIFICPP-2030
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2030
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Ferenc Gerlits
>Assignee: Ferenc Gerlits
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> (from 
> [https://github.com/apache/nifi-minifi-cpp/pull/1432#discussion_r1043352711] 
> by [~lordgamez])
> We can define processor specific metrics additionally to the general 
> processor metrics we have for all processors like transferred flow file 
> count, onTrigger calls etc. For example GetFileMetrics extends these metrics 
> with 2 additional ones.
> `in_flight_message_count_` could be added as such an additional metric for 
> PublishMQTTMetrics



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (MINIFICPP-2095) Inconsistent naming in C2 DeviceInfoNode::systemInfo::machinearch

2023-04-14 Thread Marton Szasz (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Szasz resolved MINIFICPP-2095.
-
Resolution: Fixed

> Inconsistent naming in C2 DeviceInfoNode::systemInfo::machinearch
> -
>
> Key: MINIFICPP-2095
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2095
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Martin Zink
>Assignee: Martin Zink
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Nifi and minifi java uses machineArch (the other properties also use 
> camelCase), whilst minifi cpp uses machinearch (this makes it hard to create 
> C2 servers that can be used simultaneously with java and cpp agents.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (MINIFICPP-2039) Dust off minificontroller

2023-04-14 Thread Marton Szasz (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marton Szasz resolved MINIFICPP-2039.
-
Resolution: Done

> Dust off minificontroller
> -
>
> Key: MINIFICPP-2039
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2039
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Gábor Gyimesi
>Assignee: Gábor Gyimesi
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Make sure all functionalities work properly, add test coverage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1528: MINIFICPP-2063 RFC3339 parsing with expression language

2023-04-14 Thread via GitHub


szaszm closed pull request #1528: MINIFICPP-2063 RFC3339 parsing with 
expression language
URL: https://github.com/apache/nifi-minifi-cpp/pull/1528


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1527: MINIFICPP-2030 Expose InFlightMessageCounter in PublishMQTT as processor metric

2023-04-14 Thread via GitHub


szaszm closed pull request #1527: MINIFICPP-2030 Expose InFlightMessageCounter 
in PublishMQTT as processor metric
URL: https://github.com/apache/nifi-minifi-cpp/pull/1527


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1503: MINIFICPP-2039 Dust off minificontroller

2023-04-14 Thread via GitHub


szaszm closed pull request #1503: MINIFICPP-2039 Dust off minificontroller
URL: https://github.com/apache/nifi-minifi-cpp/pull/1503


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1556: MINIFICPP-2095 fix inconsistent naming in C2 machineArch

2023-04-14 Thread via GitHub


szaszm closed pull request #1556: MINIFICPP-2095 fix inconsistent naming in C2 
machineArch
URL: https://github.com/apache/nifi-minifi-cpp/pull/1556


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Assigned] (NIFI-11433) Upgrade moment to 2.29.4

2023-04-14 Thread Scott Aslan (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-11433:
--

Assignee: Scott Aslan

> Upgrade moment to 2.29.4
> 
>
> Key: NIFI-11433
> URL: https://issues.apache.org/jira/browse/NIFI-11433
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.21.0
>Reporter: Mike R
>Assignee: Scott Aslan
>Priority: Major
>
> Upgrade moment to 2.29.4 to remediate vulnerability [Regular Expression 
> Denial of Service (ReDoS) in moment | CVE-2022-31129 | 
> Snyk|https://security.snyk.io/vuln/SNYK-JS-MOMENT-2944238]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[GitHub] [nifi] mark-bathori commented on pull request #7170: NIFI-11440: Speed up Hive Metastore based unit tests

2023-04-14 Thread via GitHub


mark-bathori commented on PR #7170:
URL: https://github.com/apache/nifi/pull/7170#issuecomment-1508531983

   Thanks @exceptionfactory for the review. I overlooked that the modified 
`AfterAll` would be still called if every test case is ignored on method level. 
I moved the `DisabledOnOs` annotation to class level.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [nifi] greyp9 commented on a diff in pull request #7173: NIFI-11439 - correct provenance reporting parameter

2023-04-14 Thread via GitHub


greyp9 commented on code in PR #7173:
URL: https://github.com/apache/nifi/pull/7173#discussion_r1166843465


##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/FetchGCSObject.java:
##
@@ -273,7 +275,15 @@ public void onTrigger(final ProcessContext context, final 
ProcessSession session
 
 final long millis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - 
startNanos);
 getLogger().info("Successfully retrieved GCS Object for {} in {} 
millis; routing to success", new Object[]{flowFile, millis});
-session.getProvenanceReporter().fetch(flowFile, "https://"; + 
bucketName + ".storage.googleapis.com/" + key, millis);
+
+String transitUri;
+try {
+final URL url = new URL(storage.getOptions().getHost());
+transitUri = String.format("%s://%s.%s/%s", url.getProtocol(), 
bucketName, url.getHost(), key);
+} catch (MalformedURLException e) {
+transitUri = e.getClass().getSimpleName();
+}

Review Comment:
   Yep, that's better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Updated] (NIFI-11449) add autocommit property to PutDatabaseRecord processor

2023-04-14 Thread Abdelrahim Ahmad (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abdelrahim Ahmad updated NIFI-11449:

Description: 
The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When 
using the processor with the Trino-JDBC-Driver or Dremio-JDBC-Driver to write 
to an Iceberg catalog, it disables the autocommit feature. This leads to errors 
such as "{*}Catalog only supports writes using autocommit: iceberg{*}".

the autocommit feature needs to be added in the processor to be 
enabled/disabled.
enabling auto-commit in the Nifi PutDatabaseRecord processor is important for 
Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by 
allowing atomic writes to be performed in the underlying database. This will 
allow the process to be widely used with bigger range of databases.

_Improving this processor will allow Nifi to be the main tool to ingest data 
into these new Technologies. So we don't have to deal with another tool to do 
so._

+*_{color:#de350b}BUT:{color}_*+



I have reviewed The {{PutDatabaseRecord}} processor in NiFi. It inserts records 
one by one into the database using a prepared statement, and commits the 
transaction at the end of the loop that processes each record. This approach 
can be inefficient and slow when inserting large volumes of data into tables 
that are optimized for bulk ingestion, such as Delta Lake, Iceberg, and Hudi 
tables.

These tables use various techniques to optimize the performance of bulk 
ingestion, such as partitioning, clustering, and indexing. Inserting records 
one by one using a prepared statement can bypass these optimizations, leading 
to poor performance and potentially causing issues such as excessive disk 
usage, increased memory consumption, and decreased query performance.

To avoid these issues, it is recommended to have a new processor, or add 
feature to the current one, to bulk insert method with AutoCommit feature when 
inserting large volumes of data into Delta Lake, Iceberg, and Hudi tables. 

 

P.S.: using PutSQL is not a have autoCommit but have the same performance 
problem described above..

Thanks and best regards :)
Abdelrahim Ahmad

  was:
The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When 
using the processor with the Trino-JDBC-Driver or Dremio-JDBC-Driver to write 
to an Iceberg catalog, it disables the autocommit feature. This leads to errors 
such as "{*}Catalog only supports writes using autocommit: iceberg{*}".

To fix this issue, the autocommit feature needs to be added in the processor to 
be enabled/disabled.
enabling auto-commit in the Nifi PutDatabaseRecord processor is important for 
Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by 
allowing atomic writes to be performed in the underlying database. This will 
allow the process to be widely used with bigger range of databases.

_*Improving this processor will allow Nifi to be the main tool to ingest data 
into these new Technologies. So we don't have to deal with another tool to do 
so.*_

P.S.: using PutSQL is not a good option at all due to the sensitivity of these 
tables when dealing with small inserts.

Thanks and best regards
Abdelrahim Ahmad


> add autocommit property to PutDatabaseRecord processor
> --
>
> Key: NIFI-11449
> URL: https://issues.apache.org/jira/browse/NIFI-11449
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.21.0
> Environment: Any Nifi Deployment
>Reporter: Abdelrahim Ahmad
>Priority: Blocker
>  Labels: Trino, autocommit, database, iceberg, putdatabaserecord
>
> The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When 
> using the processor with the Trino-JDBC-Driver or Dremio-JDBC-Driver to write 
> to an Iceberg catalog, it disables the autocommit feature. This leads to 
> errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}".
> the autocommit feature needs to be added in the processor to be 
> enabled/disabled.
> enabling auto-commit in the Nifi PutDatabaseRecord processor is important for 
> Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by 
> allowing atomic writes to be performed in the underlying database. This will 
> allow the process to be widely used with bigger range of databases.
> _Improving this processor will allow Nifi to be the main tool to ingest data 
> into these new Technologies. So we don't have to deal with another tool to do 
> so._
> +*_{color:#de350b}BUT:{color}_*+
> I have reviewed The {{PutDatabaseRecord}} processor in NiFi. It inserts 
> records one by one into the database using a prepared statement, and commits 
> the transaction at the end of the loop that processe

[jira] [Updated] (NIFI-11449) add autocommit property to PutDatabaseRecord processor

2023-04-14 Thread Abdelrahim Ahmad (Jira)


 [ 
https://issues.apache.org/jira/browse/NIFI-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abdelrahim Ahmad updated NIFI-11449:

Issue Type: New Feature  (was: Improvement)

> add autocommit property to PutDatabaseRecord processor
> --
>
> Key: NIFI-11449
> URL: https://issues.apache.org/jira/browse/NIFI-11449
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.21.0
> Environment: Any Nifi Deployment
>Reporter: Abdelrahim Ahmad
>Priority: Blocker
>  Labels: Trino, autocommit, database, iceberg, putdatabaserecord
>
> The issue is with the {{PutDatabaseRecord}} processor in Apache NiFi. When 
> using the processor with the Trino-JDBC-Driver or Dremio-JDBC-Driver to write 
> to an Iceberg catalog, it disables the autocommit feature. This leads to 
> errors such as "{*}Catalog only supports writes using autocommit: iceberg{*}".
> To fix this issue, the autocommit feature needs to be added in the processor 
> to be enabled/disabled.
> enabling auto-commit in the Nifi PutDatabaseRecord processor is important for 
> Deltalake, Iceberg, and Hudi as it ensures data consistency and integrity by 
> allowing atomic writes to be performed in the underlying database. This will 
> allow the process to be widely used with bigger range of databases.
> _*Improving this processor will allow Nifi to be the main tool to ingest data 
> into these new Technologies. So we don't have to deal with another tool to do 
> so.*_
> P.S.: using PutSQL is not a good option at all due to the sensitivity of 
> these tables when dealing with small inserts.
> Thanks and best regards
> Abdelrahim Ahmad



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (MINIFICPP-2101) Compilation failure in PutSFTPTests on MacOS

2023-04-14 Thread Martin Zink (Jira)
Martin Zink created MINIFICPP-2101:
--

 Summary: Compilation failure in PutSFTPTests on MacOS
 Key: MINIFICPP-2101
 URL: https://issues.apache.org/jira/browse/MINIFICPP-2101
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Reporter: Martin Zink
Assignee: Martin Zink


file_clock::from_sys(modification_time) used in PutSFTPTests is not available 
on every supported platform



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (MINIFICPP-1184) Cannot compile on Raspberry Pi (Raspbian bullseye)

2023-04-14 Thread Martin Zink (Jira)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Zink resolved MINIFICPP-1184.

Resolution: Fixed

this commit fixed all ARM compile issues 
[https://github.com/apache/nifi-minifi-cpp/commit/06d5467c6ca2d9a20ca9a969fd768e03025c362d]
 I was able to compile minifi on raspberry 3B+ (64 bit OS)

> Cannot compile on Raspberry Pi (Raspbian bullseye)
> --
>
> Key: MINIFICPP-1184
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1184
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Affects Versions: 0.7.0
> Environment: Raspberry Pi 4
>Reporter: dan rey
>Assignee: Dániel Bakai
>Priority: Major
>
> I am unable to compile this on a Raspberry Pi running Raspbian GNU/Linux 
> bullseye.
> I continually get an error about linking.
> [ 59%] Linking CXX executable minifi
> /usr/bin/ld: ../libminifi/libcore-minifi.a(Connection.cpp.o): in function 
> `std::__atomic_base::load(std::memory_order) const':
> /usr/include/c++/9/bits/atomic_base.h:419: undefined reference to 
> `__atomic_load_8'
> /usr/bin/ld: /usr/include/c++/9/bits/atomic_base.h:419: undefined reference 
> to `__atomic_load_8'



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request, #1558: MINIFICPP-2100 Add RockyLinux Dockerfile

2023-04-14 Thread via GitHub


lordgamez opened a new pull request, #1558:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1558

   https://issues.apache.org/jira/browse/MINIFICPP-2100
   
   
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI 
results for build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[jira] [Created] (MINIFICPP-2100) Create Dockerfile for RockyLinux build

2023-04-14 Thread Jira
Gábor Gyimesi created MINIFICPP-2100:


 Summary: Create Dockerfile for RockyLinux build
 Key: MINIFICPP-2100
 URL: https://issues.apache.org/jira/browse/MINIFICPP-2100
 Project: Apache NiFi MiNiFi C++
  Issue Type: New Feature
Reporter: Gábor Gyimesi
Assignee: Gábor Gyimesi


Due to a bug described in https://issues.apache.org/jira/browse/MINIFICPP-2077 
we cannot release convenience binaries for arm64 using the centos build. An 
alternative would be to use RockyLinux8 for this purpose, but for that we need 
to create a new Dockerfile.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[GitHub] [nifi] timeabarna commented on pull request #7092: NIFI-11327 Add Export/Import All - NiFi CLI - NiFi Registry

2023-04-14 Thread via GitHub


timeabarna commented on PR #7092:
URL: https://github.com/apache/nifi/pull/7092#issuecomment-1508199943

   Hello @bbende and @exceptionfactory,
   
   Thanks you very much for your review. I've updated the code based on your 
recommendations, improved serialisation and added storage location update.
   
   Can you please check again?
   
   Thanks in advance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org