Re: Regarding issues facing on NIFI Remote process group
Sorry for the spam, Please find the attachments here Thanks & Regards, Dharani Tirumanyam. From: Tirumanyam, Dharani Date: Monday, 1 July 2024 at 9:42 PM To: dev@nifi.apache.org Cc: Pulimi, Satish , Shankarprasad, Raghuveera , Tirumanyam, Dharani Subject: Regarding issues facing on NIFI Remote process group Hi NIFI Team, I am reaching out to you for the information regarding Remote process group. Please find the below details about NIFI and system setup we are using: NIFI version: 1.21.0 Environment details: 3 NIFI clusters for 2 NIFI servers and one NIFI client with 1TB storage. Each cluster have 3 instances of NIFI. use case: * We are having three NIFI clusters, where remote process group is running in one cluster which is RDC site(site-to-site NIFI client), SiteA (site-to-site NIFI server 1) and SiteB (site-to-site nifi server 2). * From SiteA and SiteB, only one NIFI server is reachable. If one goes down the other will starts working. Here SiteA NIFI server is currently running and RDC NIFI client is able to do data transmission between RDC NIFI client and SiteA nifi server. * When SiteA NIFI server goes down we are trying to update the SiteB nifi server URLs to expect site-to-site communication between RDC NIFI client and SiteB NIFI server. Problem statement: * Unable to stop the remote process group manually from NIFI UI to update the URLs of SiteB. * Observed once SiteA is not reachable, the remote process group in RDC NIFI client were not accepting any request ie., stop, edit, delete. Note: Three NIFI clusters are in three different environments. * Observed once we start SiteA NIFI server process we can able to stop the remote process group running in RDC client and able to update the URLs with SiteB NIFI server URLs. * If SiteA NIFI is not reachable then we are observing this issue in RDC NIFI client where unable to remote process group to update URLs. Expectation: User should able to stop, edit or delete the Remote process group running in RDC NIFI client when the NIFI server which is communicating with RDC NIFI client becomes unreachable. Please find the attachments of Remote process group highlighting the state of it. Apologise if there is any confusion created in the explanation. Need inputs to understand more to resolve the above issue. Please let us know if there is any other information needed. Thaks & Regards, Dharani Tirumanyam.
board report July 2024
Team, Thanks again for another strong quarter for NiFi. ## Description: The mission of NiFi is the creation and maintenance of software related to providing an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi MiNiFi is an edge data collection agent built to seamlessly integrate with and leverage the command and control of NiFi. There are both Java and C++ implementations. Apache NiFi Registry is a centralized registry for key configuration items including flow versions, assets, and extensions for Apache NiFi and Apache MiNiFi. Apache NiFi Nar Maven Plugin is a release artifact used for supporting the NiFi classloader isolation model. Apache NiFi Flow Design System is a theme-able set of high quality UI components and utilities for use across the various Apache NiFi web applications in order to provide a more consistent user experience. ## Project Status: Current project status: Ongoing. High. Issues for the board: None. ## Membership Data: Apache NiFi was founded 2015-07-14 (8 years ago) There are currently 67 committers and 37 PMC members in this project. The Committer-to-PMC ratio is roughly 2:1. Community changes, past quarter: - Gabor Gyimesi was added as PMC member on 2024-06-26. - Lehel Boer was added as committer on 2024-06-16. ## Project Activity: We released Apache NiFi 2.0.0 milestone 4 (M4) on July 1st 2024. M4 contains more than 150 JIRAs including moving completely to the new UI and providing more integration options for Kafka, splitting packet capture data, and a new Python source component API. We released the NiFi NAR Maven Plugin 2.0.0 on May 28 2024 supporting the new rules extension API in build manifests. We released the MiNiFi CPP 0.99.0 on May 18 2024 including many improvements and MacOS service integration, SMB, Grafana, Jolt transforms for JSON data, greater efficiency when merging larger datasets with MergeContent. We released NiFi 2.0.0 milestone 3 (M3) on May 16 2024. M3 included more than 400 JIRAs with better integrations to AWS, Azure, OpenSearch and SMB. The new UI is now available and considered feature complete but the old UI remains available. ## Community Health: Community health remains strong and growing. JIRA and Mailing list activity remains consistent. The slack community grew by another 150 participants in the general channel alone, increasing from 3,133 to 3,284 since the last report. We continue to produce frequent releases in the NiFi community and enjoy rather active vote participation even when it takes a few release candidates to get it right. We are seeing 30 or more unique contributors monthly with on average several commits per day landing and that is just the core nifi codebase. Code activity for NiFi this calendar year appears higher than any previous year when reviewing Github activity data.
[ANNOUNCE] Apache NiFi 2.0.0-M4 Released
The Apache NiFi Team is pleased to announce the release of Apache NiFi 2.0.0-M4. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. https://nifi.apache.org The release artifacts can be downloaded from the project website. https://nifi.apache.org/download/ Maven artifacts have been released and mirrored according to ASF artifact distribution processes. Issues resolved in Apache NiFi 2.0.0-M4 are listed in Jira Release Notes. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12354678 Highlights of the release are available on the project wiki page: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M4 Regards, The Apache NiFi Team
Regarding issues facing on NIFI Remote process group
Hi NIFI Team, I am reaching out to you for the information regarding Remote process group. Please find the below details about NIFI and system setup we are using: NIFI version: 1.21.0 Environment details: 3 NIFI clusters for 2 NIFI servers and one NIFI client with 1TB storage. Each cluster have 3 instances of NIFI. use case: * We are having three NIFI clusters, where remote process group is running in one cluster which is RDC site(site-to-site NIFI client), SiteA (site-to-site NIFI server 1) and SiteB (site-to-site nifi server 2). * From SiteA and SiteB, only one NIFI server is reachable. If one goes down the other will starts working. Here SiteA NIFI server is currently running and RDC NIFI client is able to do data transmission between RDC NIFI client and SiteA nifi server. * When SiteA NIFI server goes down we are trying to update the SiteB nifi server URLs to expect site-to-site communication between RDC NIFI client and SiteB NIFI server. Problem statement: * Unable to stop the remote process group manually from NIFI UI to update the URLs of SiteB. * Observed once SiteA is not reachable, the remote process group in RDC NIFI client were not accepting any request ie., stop, edit, delete. Note: Three NIFI clusters are in three different environments. * Observed once we start SiteA NIFI server process we can able to stop the remote process group running in RDC client and able to update the URLs with SiteB NIFI server URLs. * If SiteA NIFI is not reachable then we are observing this issue in RDC NIFI client where unable to remote process group to update URLs. Expectation: User should able to stop, edit or delete the Remote process group running in RDC NIFI client when the NIFI server which is communicating with RDC NIFI client becomes unreachable. Please find the attachments of Remote process group highlighting the state of it. Apologise if there is any confusion created in the explanation. Need inputs to understand more to resolve the above issue. Please let us know if there is any other information needed. Thaks & Regards, Dharani Tirumanyam.
[RESULT][VOTE] Release Apache NiFi 2.0.0-M4 (RC1)
Apache NiFi Community, I am pleased to announce that the 2.0.0-M4 release of Apache NiFi passes: 6 +1 (binding) votes 4 +1 (non-binding) votes 0 0 votes 0 -1 votes Thanks to all who helped make this release possible! The release artifacts will be published in the next 24 hours. Here is the vote thread: https://lists.apache.org/thread/jdb2j398ow6221rx4og7p958nnbj7bs9 Regards, David Handermann
Re: [VOTE] Release Apache NiFi 2.0.0-M4 (RC1)
+1 (non-binding) - went through the helper guide, - did a full build with contrib check - verified signatures and hashes Environment: - macOS Version 14.3.1 - OpenJDK Runtime Environment Zulu21.28+85-CA (build 21+35) - Apache Maven 3.9.6 Created and ran some simple flows. While testing NIFI-13356 I found a new issue that wasn't covered in the ticket. The value parse can fail when coercing types of repeated fields. Since it's a new issue it shouldn't block the release. I'll create a ticket and fix this. Thanks for RMing David! Regards, Mark David Handermann ezt írta (időpont: 2024. júl. 1., H, 15:42): > +1 (binding) > > On Fri, Jun 28, 2024 at 8:40 AM David Handermann > wrote: > > > > Team, > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi 2.0.0-M4. > > > > Please review the following guide for how to verify a release candidate > build: > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification > > > > The source being voted on the and the convenience binaries are > > available on the Apache Distribution Repository: > > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M4 > > > > The build artifacts are available on the Apache Nexus Repository: > > > > https://repository.apache.org/content/repositories/orgapachenifi-1275 > > > > Git Tag: nifi-2.0.0-M4-RC1 > > Git Commit ID: 19c5be01d463bc38ec0f5008549a2a42e589436d > > GitHub Commit Link: > > > https://github.com/apache/nifi/commit/19c5be01d463bc38ec0f5008549a2a42e589436d > > > > Checksums of nifi-2.0.0-M4-source-release.zip > > > > SHA256: d882f05cec09ee1bfafaa3d4cde8f8660512d09765b5c400471f3a6e014029a6 > > SHA512: > d429cd67fb0b7d9737c59cb834106d7b6e25cbdb91e3ecc5290be865a1313cbebbc314c2e0e54228226f021c44f0a86c745a18c148247c632a739c871c5fa013 > > > > Release artifacts are signed with the following key: > > > > https://people.apache.org/keys/committer/exceptionfactory.asc > > > > KEYS file is available on the Apache Distribution Repository: > > > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > Issues resolved for this version: 153 > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12354678 > > > > Release note highlights can be found on the project wiki: > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M4 > > > > The vote will be open for 72 hours. > > > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build from source, and test. > > Then please vote: > > > > [] +1 Release this package as nifi-2.0.0-M4 > > [] +0 no opinion > > [] -1 Do not release this package because... >
Re: [VOTE] Release Apache NiFi 2.0.0-M4 (RC1)
+1 (binding) On Fri, Jun 28, 2024 at 8:40 AM David Handermann wrote: > > Team, > > I am pleased to be calling this vote for the source release of Apache > NiFi 2.0.0-M4. > > Please review the following guide for how to verify a release candidate build: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification > > The source being voted on the and the convenience binaries are > available on the Apache Distribution Repository: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M4 > > The build artifacts are available on the Apache Nexus Repository: > > https://repository.apache.org/content/repositories/orgapachenifi-1275 > > Git Tag: nifi-2.0.0-M4-RC1 > Git Commit ID: 19c5be01d463bc38ec0f5008549a2a42e589436d > GitHub Commit Link: > https://github.com/apache/nifi/commit/19c5be01d463bc38ec0f5008549a2a42e589436d > > Checksums of nifi-2.0.0-M4-source-release.zip > > SHA256: d882f05cec09ee1bfafaa3d4cde8f8660512d09765b5c400471f3a6e014029a6 > SHA512: > d429cd67fb0b7d9737c59cb834106d7b6e25cbdb91e3ecc5290be865a1313cbebbc314c2e0e54228226f021c44f0a86c745a18c148247c632a739c871c5fa013 > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/exceptionfactory.asc > > KEYS file is available on the Apache Distribution Repository: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > Issues resolved for this version: 153 > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12354678 > > Release note highlights can be found on the project wiki: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M4 > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. > Then please vote: > > [] +1 Release this package as nifi-2.0.0-M4 > [] +0 no opinion > [] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 2.0.0-M4 (RC1)
+1 (binding) - Verified hashes and signatures - Successfully built NiFi 2.0.0-M4 with contrib-check in the following environment: - Ubuntu 22.04 6.5.0-41-generic - java version "21" 2023-09-19 LTS - Apache Maven 3.9.6 - Tested a simple flow using a python AI flowfiletransform processor - Verified NIFI-13427: Python API extended to support source processors - tested a flow with a source python processor - Tested S2S communication and HTTP data forwarding from MiNiFi C++ Thanks David for RMing! Best Regards, Gabor On Mon, 1 Jul 2024 at 10:41, Ferenc Kis wrote: > > +1 (non-binding) > > Went through the helper guide, local maven repo cleaned up, full clean > build with contrib check, verified signatures and hashes > > Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae) > Maven home: /Users/fkis/.sdkman/candidates/maven/current > Java version: 21.0.3, vendor: Azul Systems, Inc., runtime: > /Users/fkis/.sdkman/candidates/java/21.0.3-zulu/zulu-21.jdk/Contents/Home > Default locale: en_HU, platform encoding: UTF-8 > OS name: "mac os x", version: "14.5", arch: "aarch64", family: "mac" > > Validations performed: > - Started NiFi, created a simple flow with ListenHTTP and Input Port. > Validated ListenHTTP processor is able to receive data > - Started MiNiFi Java: > * created a simple GenerateFlowFile -> InvokeHttp flow and > GenerateFlowFile -> RemoteProcessGroup flow and pushed to MiNiFi via C2 > protocol > * Validated that connectivity between NiFi and MiNiFi works via both > InvokeHTTP and S2S > - NIFI-13395 Add python processors to MiNiFi: included Python processors > from the extension repo. Processors were included in the manifest > - NIFI-13396 Include python processors' allowable values in manifest: > values were correctly sent via the manifest > - NIFI-13242 Sync Resource command: synced (added and deleted) several > resources with C2 server - both asset and extension type > - NIFI-13450 Regenerate MiNiFi Agent Manifest between Heartbeats: manifest > is refreshed for each heartbeat correctly > - NIFI-13407 Sensible defaults for Manifest Id and Runtime Type in MiNiFi > bootstrap: parameters now got sensible values instead of empty string > > Thanks for RMing David! > > Regards > Ferenc > > On Sat, Jun 29, 2024 at 10:02 PM Csaba Bejan wrote: > > > +1 (binding) > > > >- Went through the helper guide and did a clean build > >- Verified signatures and hashes > >- Built on OSX 14.2.1 > >- Java version: Zulu21.28+85-CA (build 21+35) > >- Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546) > >- Started NiFi and created a simple flow, everything looks good > >- Started MiNiFi and verified integration with C2 Server. Played around > >with the C2 protocol > > > > Also verified: > > > >- NIFI-13407 Sensible defaults for Manifest Id and Runtime Type in > >MiNiFi bootstrap > >- NIFI-13450 Regenerate MiNiFi Agent Manifest between Heartbeats > >- It is not a release blocker just wanted to note that if c2 is enabled > > as manifest is regenerated for each HB at every HB I'm getting > > warnings > > which can be quite noisy > > 2024-06-29 15:37:34,920 WARN [pool-16-thread-1] > > o.a.n.m.StandardRuntimeManifestService There is no extension > > manifest for > > bundle [org.apache.nifi:python-extensions:unversioned] > > 2024-06-29 15:37:34,921 WARN [pool-16-thread-1] > > o.a.n.m.StandardRuntimeManifestService There is no extension > > manifest for > > bundle [default:system:unversioned] > >- NIFI-13436 Retain queue for non-modified connections during MiNiFi > >flow update > >- NIFI-13309 Lookup compatible bundles even if previous flow was empty > >- NIFI-13395 Add python processors to MiNiFi > > - Python processors are showing up under minifi now when enabled and > > dropped in the extension directory > > > > Thanks for RMing, David! > > > > Regards, > > > > Csaba > > > > On Sat, Jun 29, 2024 at 11:30 AM Joe Witt wrote: > > > > > +1 binding > > > > > > full clean build w contrib > > > full suite of integration tests including python/etc.. (see below) > > > built docker image and deployed over an existing set of volumes showing > > > upgrades well from previous releases > > > updated release notes a bit to focus more on end user perspective > > > > > > Test error during integration tests suggesting potentially unstable test. > > > Ran fine on second try > > > [ERROR] Tests run: 33, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > > > 0.657 s <<< FAILURE! -- in > > > org.apache.nifi.processors.standard.util.TestServerSFTPTransfer > > > [ERROR] > > > > > > > > org.apache.nifi.processors.standard.util.TestServerSFTPTransfer.testGetListingWithBatchSize > > > -- Time elapsed: 0.019 s <<< ERROR! > > > org.apache.nifi.processors.standard.socket.ClientConnectException: SSH > > > Client connection failed [127.0.0.1:51893] > > > at > > > > > > > >
Re: [VOTE] Release Apache NiFi 2.0.0-M4 (RC1)
+1 (non-binding) Went through the helper guide, local maven repo cleaned up, full clean build with contrib check, verified signatures and hashes Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae) Maven home: /Users/fkis/.sdkman/candidates/maven/current Java version: 21.0.3, vendor: Azul Systems, Inc., runtime: /Users/fkis/.sdkman/candidates/java/21.0.3-zulu/zulu-21.jdk/Contents/Home Default locale: en_HU, platform encoding: UTF-8 OS name: "mac os x", version: "14.5", arch: "aarch64", family: "mac" Validations performed: - Started NiFi, created a simple flow with ListenHTTP and Input Port. Validated ListenHTTP processor is able to receive data - Started MiNiFi Java: * created a simple GenerateFlowFile -> InvokeHttp flow and GenerateFlowFile -> RemoteProcessGroup flow and pushed to MiNiFi via C2 protocol * Validated that connectivity between NiFi and MiNiFi works via both InvokeHTTP and S2S - NIFI-13395 Add python processors to MiNiFi: included Python processors from the extension repo. Processors were included in the manifest - NIFI-13396 Include python processors' allowable values in manifest: values were correctly sent via the manifest - NIFI-13242 Sync Resource command: synced (added and deleted) several resources with C2 server - both asset and extension type - NIFI-13450 Regenerate MiNiFi Agent Manifest between Heartbeats: manifest is refreshed for each heartbeat correctly - NIFI-13407 Sensible defaults for Manifest Id and Runtime Type in MiNiFi bootstrap: parameters now got sensible values instead of empty string Thanks for RMing David! Regards Ferenc On Sat, Jun 29, 2024 at 10:02 PM Csaba Bejan wrote: > +1 (binding) > >- Went through the helper guide and did a clean build >- Verified signatures and hashes >- Built on OSX 14.2.1 >- Java version: Zulu21.28+85-CA (build 21+35) >- Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546) >- Started NiFi and created a simple flow, everything looks good >- Started MiNiFi and verified integration with C2 Server. Played around >with the C2 protocol > > Also verified: > >- NIFI-13407 Sensible defaults for Manifest Id and Runtime Type in >MiNiFi bootstrap >- NIFI-13450 Regenerate MiNiFi Agent Manifest between Heartbeats >- It is not a release blocker just wanted to note that if c2 is enabled > as manifest is regenerated for each HB at every HB I'm getting > warnings > which can be quite noisy > 2024-06-29 15:37:34,920 WARN [pool-16-thread-1] > o.a.n.m.StandardRuntimeManifestService There is no extension > manifest for > bundle [org.apache.nifi:python-extensions:unversioned] > 2024-06-29 15:37:34,921 WARN [pool-16-thread-1] > o.a.n.m.StandardRuntimeManifestService There is no extension > manifest for > bundle [default:system:unversioned] >- NIFI-13436 Retain queue for non-modified connections during MiNiFi >flow update >- NIFI-13309 Lookup compatible bundles even if previous flow was empty >- NIFI-13395 Add python processors to MiNiFi > - Python processors are showing up under minifi now when enabled and > dropped in the extension directory > > Thanks for RMing, David! > > Regards, > > Csaba > > On Sat, Jun 29, 2024 at 11:30 AM Joe Witt wrote: > > > +1 binding > > > > full clean build w contrib > > full suite of integration tests including python/etc.. (see below) > > built docker image and deployed over an existing set of volumes showing > > upgrades well from previous releases > > updated release notes a bit to focus more on end user perspective > > > > Test error during integration tests suggesting potentially unstable test. > > Ran fine on second try > > [ERROR] Tests run: 33, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > > 0.657 s <<< FAILURE! -- in > > org.apache.nifi.processors.standard.util.TestServerSFTPTransfer > > [ERROR] > > > > > org.apache.nifi.processors.standard.util.TestServerSFTPTransfer.testGetListingWithBatchSize > > -- Time elapsed: 0.019 s <<< ERROR! > > org.apache.nifi.processors.standard.socket.ClientConnectException: SSH > > Client connection failed [127.0.0.1:51893] > > at > > > > > org.apache.nifi.processors.standard.ssh.StandardSSHClientProvider.getClient(StandardSSHClientProvider.java:116) > > at > > > > > org.apache.nifi.processors.standard.util.SFTPTransfer.getSFTPClient(SFTPTransfer.java:631) > > at > > > > > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:327) > > at > > > > > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:289) > > at > > > > > org.apache.nifi.processors.standard.util.TestServerSFTPTransfer.testGetListingWithBatchSize(TestServerSFTPTransfer.java:225) > > at java.base/java.lang.reflect.Method.invoke(Method.java:580) > > at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) > > at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) > > Caused by:
