Re: [VOTE] Release Apache NiFi 1.27.0 (RC2)
+1 (non-binding) checksums are good Compiled with Apache Maven 3.9.6 1.8.0_412 (openjdk) RHEL 8 Ran some basic flows Upgraded registry (with pre-existing versioned flows) from 1.26 to 1.27 successfully. Created new PGs from versioned flows. Pushed new versions of flows to registry. Discovered issue where nifi will not recognize versioned PG has been modified if modification was done to versioned inner PG. Created https://issues.apache.org/jira/browse/NIFI-13486 This is not a blocker level issue (and further review found it existed in 1.26 as well.) Thank you Pierre and team for the upcoming release. Thanks, Nissim Shiman On Wednesday, July 3, 2024 at 12:42:40 PM UTC, Pierre Villard wrote: Team, I am pleased to be calling this vote for the source release of Apache NiFi 1.27.0. Please review the following guide for how to verify a release candidate build: https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification The source being voted on the and the convenience binaries are available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/dev/nifi/nifi-1.27.0 The build artifacts are available on the Apache Nexus Repository: https://repository.apache.org/content/repositories/orgapachenifi-1276 Git Tag: nifi-1.27.0-RC2 Git Commit ID: e0c4461d90bd4f6e5f2b81765bcff5cd97ed3e18 GitHub Commit Link: https://github.com/apache/nifi/commit/e0c4461d90bd4f6e5f2b81765bcff5cd97ed3e18 Checksums of nifi-1.27.0-source-release.zip SHA256: 0ac01d54eeceb4a4f4863880bf67dfa71e6a585036c5caf0546c592bf55ced48 SHA512: 675c7750752bf3092061c9eaac39a975955e9bf881e6bee3124c5842738d8c8626b6a551f33ef7a678018bd83e0323c1f0f0d79101255494d8ca91be7fc750f5 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pvillard.asc KEYS file is available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/release/nifi/KEYS Issues resolved for this version: 37 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354832 Release note highlights can be found on the project wiki: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.27.0 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [] +1 Release this package as nifi-1.27.0 [] +0 no opinion [] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 2.0.0-M4 (RC1)
+1 (non-binding) checksums look good Compiled withApache Maven 3.9.8 Java 21.0.3 (openjdk) RHEL 8 Set up/ran site-to-site flow and some rest api calls Thank you David and team, Nissim Shiman On Friday, June 28, 2024 at 09:40:57 AM EDT, David Handermann wrote: Team, I am pleased to be calling this vote for the source release of Apache NiFi 2.0.0-M4. Please review the following guide for how to verify a release candidate build: https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification The source being voted on the and the convenience binaries are available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M4 The build artifacts are available on the Apache Nexus Repository: https://repository.apache.org/content/repositories/orgapachenifi-1275 Git Tag: nifi-2.0.0-M4-RC1 Git Commit ID: 19c5be01d463bc38ec0f5008549a2a42e589436d GitHub Commit Link: https://github.com/apache/nifi/commit/19c5be01d463bc38ec0f5008549a2a42e589436d Checksums of nifi-2.0.0-M4-source-release.zip SHA256: d882f05cec09ee1bfafaa3d4cde8f8660512d09765b5c400471f3a6e014029a6 SHA512: d429cd67fb0b7d9737c59cb834106d7b6e25cbdb91e3ecc5290be865a1313cbebbc314c2e0e54228226f021c44f0a86c745a18c148247c632a739c871c5fa013 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/exceptionfactory.asc KEYS file is available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/release/nifi/KEYS Issues resolved for this version: 153 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354678 Release note highlights can be found on the project wiki: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M4 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [] +1 Release this package as nifi-2.0.0-M4 [] +0 no opinion [] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 1.27.0 (RC1)
Team, I am running into difficulty starting nifi registry. It was compiled (and is being run) with java 1.8 (openjdk-1.8.0.412) on RHEL 8 nifi itself compiles and runs fine. I was wondering is other people have run into anything similar. The error is (from nifi-registry-app.log): 2024-06-28 14:44:33,150 WARN [main] d.o.a.n.r.bootstrap.RunNiFiRegistry Support for Java 8 is deprecated. Java 11 is the minimum recommended versionorg.apache.nifi.deprecation.log.DeprecationException: Reference Class [org.apache.nifi.registry.bootstrap.RunNiFiRegistry] ClassLoader [sun.misc.Launcher$AppClassLoader@232204a1] at org.apache.nifi.deprecation.log.StandardDeprecationLogger.getExtendedArguments(StandardDeprecationLogger.java:63) at org.apache.nifi.deprecation.log.StandardDeprecationLogger.warn(StandardDeprecationLogger.java:54) at org.apache.nifi.registry.bootstrap.RunNiFiRegistry.start(RunNiFiRegistry.java:941) at org.apache.nifi.registry.bootstrap.RunNiFiRegistry.main(RunNiFiRegistry.java:206)2024-06-28 14:44:33,506 INFO [main] org.apache.nifi.registry.NiFiRegistry Launching NiFi Registry...2024-06-28 14:44:33,508 INFO [main] org.apache.nifi.registry.NiFiRegistry Read property protection key from /registry_1_27_0_RC1/nifi-registry-1.27.0/conf/bootstrap.conf2024-06-28 14:44:33,525 WARN [main] o.a.n.p.AbstractBootstrapPropertiesLoader No encryption key present in the bootstrap.conf file at /registry_1_27_0_RC1/nifi-registry-1.27.0/conf/bootstrap.conf and (nifi-registry-bootstrap.log) 2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/nifi/deprecation/log/DeprecationLoggerFactory2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr at org.apache.nifi.registry.properties.NiFiRegistryPropertiesLoader.(NiFiRegistryPropertiesLoader.java:38)2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr at java.lang.Class.forName0(Native Method)2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr at java.lang.Class.forName(Class.java:348)2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr at org.apache.nifi.registry.NiFiRegistry.initializeProperties(NiFiRegistry.java:189)2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr at org.apache.nifi.registry.NiFiRegistry.main(NiFiRegistry.java:153)2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Caused by: java.lang.ClassNotFoundException: org.apache.nifi.deprecation.log.DeprecationLoggerFactory2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr at java.net.URLClassLoader.findClass(URLClassLoader.java:387)2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr at java.lang.ClassLoader.loadClass(ClassLoader.java:418)2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr at java.lang.ClassLoader.loadClass(ClassLoader.java:351)2024-06-28 14:44:34,477 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr ... 5 more2024-06-28 14:44:35,174 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry NiFi Registry never started. Will not restart NiFi Registry Thanks, Nissim Shiman On Thursday, June 27, 2024 at 09:01:30 PM UTC, Pierre Villard wrote: Team, I am pleased to be calling this vote for the source release of Apache NiFi 1.27.0. Please review the following guide for how to verify a release candidate build: https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification The source being voted on the and the convenience binaries are available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/dev/nifi/nifi-1.27.0 The build artifacts are available on the Apache Nexus Repository: https://repository.apache.org/content/repositories/orgapachenifi-1272 Git Tag: nifi-1.27.0-RC1 Git Commit ID: bd86d1f07fe8d0f45b72e91e925ee85ffa7ed143 GitHub Commit Link: https://github.com/apache/nifi/commit/bd86d1f07fe8d0f45b72e91e925ee85ffa7ed143 Checksums of nifi-1.27.0-source-release.zip SHA256: 7df2933b366aff8e1dc3abe9bafb3e8891186ca937d34d646952d43db624d4e2 SHA512: 36c0f107b5a98388c2fb2c48d24f4df9980d9685a9b31d08c0a7a6a47b54de556ff368a8c622a5011dab4448274fc16f6e18c5d29368a81d003baa7c7382bd4d Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pvillard.asc KEYS file is available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/release/nifi/KEYS Issues resolved for this version: 35 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354832 Release note highlights can be found on the project wiki: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.27.0 Th
Re: [VOTE] Release Apache NiFi 1.26.0 (RC1)
+1 (non-binding) verified source release sha256/512 checksums successfully ran build using: Apache Maven 3.9.6 Java openjdk version: 1.8.0_402 linux kernel 3.10.0-1160 Ran various simple flows successfully. Tested against out of the box/empty registry, (i.e. created bucket, imported PG. made new version of PG, imported new version of PG back to graph) Migrated existing/populated registry from 1.24.0 to this RC successfully. (i.e. and tested importing PGs, committing new versions of PGs to registry successfully, importing updated PG to graph) Tested/verified: NIFI-12858 - Processor change history on property hover Thank you Pierre and team for 1.26.0! Nissim Shiman On Friday, May 3, 2024 at 12:47:45 PM UTC, Pierre Villard wrote: Team, I am pleased to be calling this vote for the source release of Apache NiFi 1.26.0. Please review the following guide for how to verify a release candidate build: https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification The source being voted on the and the convenience binaries are available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/dev/nifi/nifi-1.26.0 The build artifacts are available on the Apache Nexus Repository: https://repository.apache.org/content/repositories/orgapachenifi-1265 Git Tag: nifi-1.26.0-RC1 Git Commit ID: 338083f6850b61cd2651e41bbd73b3cb5330d98c GitHub Commit Link: https://github.com/apache/nifi/commit/338083f6850b61cd2651e41bbd73b3cb5330d98c Checksums of nifi-1.26.0-source-release.zip SHA256: 75ea201c12bb99672b1f075c9520b5f7df8b24e033ec47121848b351a0d47790 SHA512: 5b754d899ce8cff900d5871d44c2fda9224e69fe8a0fe9a7121f3359ed8881300e4d4d0b2fe5befea276e0495c7ebbed04cc307c18527aa7a1cea25a923a Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pvillard.asc KEYS file is available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/release/nifi/KEYS Issues resolved for this version: 128 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354185 Release note highlights can be found on the project wiki: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.26.0 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [] +1 Release this package as nifi-1.26.0 [] +0 no opinion [] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC4)
+1 (non-binding) verified source release sha256/512 checksums successfully ran build using: Apache Maven 3.9.6 Java 21 2023-09-19 LTS linux kernel 3.10.0-1160 Ran various simple flows successfully. Migrated registry from 1.24 and 2.0.0-M1 to this RC successfully. (i.e. and tested importing PGs, committing new versions of PGs to registry successfully, both from nifi and filesystem). Verified flows were persisted in registry after registry restart as well. Noticed non-blocking issue where reporting tasks' link to referenced controller service is not completely working. Created NIFI-12678 for it. Tested: NIFI-11389 Controller Services's link to referencing Controller Services components not always working NIFI-12387 Flow Configuration History can record a Comments change for Controller Service when no changes have been made NIFI-12394 when importing versioned flow with component that migrates properties, controller service reference is invalid NIFI-12666 Correct NiFi Registry Data Source Configuration Thank you David and team for the upcoming release! Nissim Shiman On Friday, January 26, 2024 at 02:30:45 AM UTC, David Handermann wrote: Team, I am pleased to be calling this vote for the source release of Apache NiFi 2.0.0-M2. Please review the following guide for how to verify a release candidate build: https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification The source being voted on and the convenience binaries are available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2 The build artifacts are available on the Apache Nexus Repository: https://repository.apache.org/content/repositories/orgapachenifi-1264 Git Tag: nifi-2.0.0-M2-RC4 Git Commit ID: 640b7bdfbbb8842f057a9bf49dc2b9b5d092abda GitHub Commit Link: https://github.com/apache/nifi/commit/640b7bdfbbb8842f057a9bf49dc2b9b5d092abda Checksums of nifi-2.0.0-M2-source-release.zip SHA256: 1946eceb3ae4f541545e93ddc6dd2cbe2a3302a6a46e6c584f3ffc1c1bd1e18b SHA512: e8e67e1e67359553479c0721a1c49ae6706cc6882eadf92e1f5ccc2619637ab87119a06221980d4c34d99b7b6d9a2138c77440b407074090e727b5d4447ab799 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/exceptionfactory.asc KEYS file is available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/release/nifi/KEYS Issues resolved for this version: 214 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353861 Release note highlights can be found on the project wiki: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [] +1 Release this package as nifi-2.0.0-M2 [] +0 no opinion [] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 2.0.0-M2 (RC2)
Team, I'm having difficulty setting up the registry for this RC and was wondering if anyone has a similar situation. The registry starts fine, new buckets an be added, flows can be imported, but they don't seem to persist on a restart. Also, having difficulty migrating registry from 1.24.0 or 2.0.0 M1 to this RC as well (i.e. the registry starts up, but doesn't see any of the buckets/versioned flows) Thanks, Nissim Shiman On Wednesday, January 24, 2024 at 10:35:34 AM EST, David Handermann wrote: Team, I am pleased to be calling this vote for the source release of Apache NiFi 2.0.0-M2. Please review the following guide for how to verify a release candidate build: https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification The source being voted on and the convenience binaries are available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M2 The build artifacts are available on the Apache Nexus Repository: https://repository.apache.org/content/repositories/orgapachenifi-1259 Git Tag: nifi-2.0.0-M2-RC2 Git Commit ID: f50ab61772de816b08edffc97393a856b0a87ed2 GitHub Commit Link: https://github.com/apache/nifi/commit/f50ab61772de816b08edffc97393a856b0a87ed2 Checksums of nifi-2.0.0-M2-source-release.zip SHA256: ac0a96254f5f5f4662ee7e75eb7e6b510ea724e673feda8352d026ba21681de4 SHA512: 75826a049325fa3cbabd96b6a4f259b16499102cf1880b688bbef0824196e499f146c5141a821bb1543d6fed5d02d1db1c653f628580ebeeae18917aeaf05ed7 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/exceptionfactory.asc KEYS file is available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/release/nifi/KEYS Issues resolved for this version: 205 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353861 Release note highlights can be found on the project wiki: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M2 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [] +1 Release this package as nifi-2.0.0-M2 [] +0 no opinion [] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 2.0.0-M1 (RC5)
+1 (non-binding) verified source release sha256/512 checksums successfully ran build using: Apache Maven 3.9.5 Java 21 2023-09-19 LTS linux kernel 3.10.0-1160 Ran various simple flows successfully. Migrated registry from 1.23.2 to this RC successfully. (i.e. and tested importing PGs, committing new versions of PGs to registry successfully.) Tested: NIFI-10950 Distribute Load processor - remove Load Distribution Service as Distribution Strategy NIFI-11463 Add option for IdentifyMimeType custom mime definitions to be in addition to the default tika definitions NIFI-12301 When implementing migrateProperties, use of PropertyConfiguration.hasProperty() may return wrong result. (I find migrateProperties really useful. Thank you, markap14, for this feature) NIFI-12315 Difficulty importing PG from Registry following deprecation removals Thank you David and team for the upcoming release! Nissim Shiman On Tuesday, November 21, 2023 at 10:40:25 PM UTC, David Handermann wrote: Team, I am pleased to be calling this vote for the source release of Apache NiFi 2.0.0-M1. Please review the following guide for how to verify a release candidate build: https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification The source being voted on the and the convenience binaries are available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M1 The build artifacts are available on the Apache Nexus Repository: https://repository.apache.org/content/repositories/orgapachenifi-1248 Git Tag: nifi-2.0.0-M1-RC5 Git Commit ID: f81a3597d40f3fe6df93d55347a11474ee6af2c8 GitHub Commit Link: https://github.com/apache/nifi/commit/f81a3597d40f3fe6df93d55347a11474ee6af2c8 Checksums of nifi-2.0.0-M1-source-release.zip SHA256: 217096b30c211b8fb6bd069832416b507b258f335d7301772a8c047f470a4b30 SHA512: 949849c59d2d8a2d0a441fd14c718f97cfb86c72fdd2de9a954a5829eebad6f9ad60622ecc6db200e246f77f771530be20f47793b3a8a18d2ceed252223139ee Release artifacts are signed with the following key: https://people.apache.org/keys/committer/exceptionfactory.asc KEYS file is available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/release/nifi/KEYS Issues resolved for this version: 958 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12339599 Release note highlights can be found on the project wiki: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [] +1 Release this package as nifi-2.0.0-M1 [] +0 no opinion [] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 2.0.0-M1 (RC5)
Hi David,
This makes sense. Thank you very much!
Nissim
On Wednesday, November 22, 2023 at 01:14:53 PM EST, David Handermann
wrote:
Hi Nissim,
Thanks for raising this question.
The short answer is that this is expected behavior for NiFi and NiFi Registry.
NiFi and NiFI Registry include automated migration capabilities for
the H2 database in version 1. This allows straightforward upgrades
within the same major release version.
NiFi 2.0.0 removes this automated migration capability, and it is
expected that any upgrades from version 1 will require first upgrading
to the latest minor release version, which will be 1.24.0 following
the completion of this release cycle.
NiFi Registry 1.24.0 and 2.0.0-M1 will continue to support H2, but
version 2.0.0 will require upgrading to 1.24.0 before upgrading to
2.0.0.
Regards,
David Handermann
On Wed, Nov 22, 2023 at 12:03 PM Nissim Shiman
wrote:
>
> Team,
>
>
> I'm having difficulty migrating from registry 1.22.0 or earlier to 2.0.0-M1.
> This was not seen on when moving to 1.24.0 (RC3).
>
>
> i.e. Migrating registry from 1.23.2 to 2.0.0-M1 or 1.24.0 (RC3) works fine.
>
>
> ("Migrating" meaning a copying of the database and flow_storage directories
> from the root directory of the older registry to 2.0.0-M1/1.24.0 (RC3))
>
>
> Registry is having difficulty starting up. Has anyone else noticed this?
>
>
> Caused by: org.h2.jdbc.JdbcSQLNonTransientConnectionException: Unsupported
> database file version or invalid file header in file "/ directory>/registry_2_0_0_M1/nifi-registry-2.0.0-M1/database/nifi-registry-primary.mv.db"
> [90048-224] at
> org.h2.message.DbException.getJdbcSQLException(DbException.java:690)
> at org.h2.message.DbException.getJdbcSQLException(DbException.java:489)
> at org.h2.message.DbException.get(DbException.java:212) at
> org.h2.mvstore.db.Store.convertMVStoreException(Store.java:158) at
> org.h2.mvstore.db.Store.(Store.java:142) at
> org.h2.engine.Database.(Database.java:326) at
> org.h2.engine.Engine.openSession(Engine.java:92) at
> org.h2.engine.Engine.openSession(Engine.java:222) at
> org.h2.engine.Engine.createSession(Engine.java:201) at
> org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:343)
> at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:125) at
> org.h2.Driver.connect(Driver.java:59) at
> com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
> at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359)
> at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)
> at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470)
> at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)
> at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:100)
> at
> com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
> at
> org.apache.nifi.registry.db.CustomFlywayConfiguration.getDatabaseType(CustomFlywayConfiguration.java:91)
> ... 121 common frames omittedCaused by:
> org.h2.mvstore.MVStoreException: The write format 2 is smaller than the
> supported format 3 [2.2.224/5] at
> org.h2.mvstore.DataUtils.newMVStoreException(DataUtils.java:996) at
> org.h2.mvstore.FileStore.getUnsupportedWriteFormatException(FileStore.java:943)
> at
> org.h2.mvstore.FileStore.processCommonHeaderAttributes(FileStore.java:547)
> at
> org.h2.mvstore.RandomAccessStore.readStoreHeader(RandomAccessStore.java:227)
> at org.h2.mvstore.FileStore.start(FileStore.java:916) at
> org.h2.mvstore.MVStore.(MVStore.java:289) at
> org.h2.mvstore.MVStore$Builder.open(MVStore.java:2035) at
> org.h2.mvstore.db.Store.(Store.java:133) ... 136 common frames
> omitted
>
> Thanks,
> Nissim Shiman
> On Tuesday, November 21, 2023 at 10:40:25 PM UTC, David Handermann
> wrote:
>
> Team,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 2.0.0-M1.
>
> Please review the following guide for how to verify a release candidate build:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
>
> The source being voted on the and the convenience binaries are
> available on the Apache Distribution Repository:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M1
>
> The build artifacts are available on the Apache Nexus Repository:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1248
>
> Git Tag: nifi-2.0.0-M1-RC5
> Git Commit ID: f8
Re: [VOTE] Release Apache NiFi 2.0.0-M1 (RC5)
Team,
I'm having difficulty migrating from registry 1.22.0 or earlier to 2.0.0-M1.
This was not seen on when moving to 1.24.0 (RC3).
i.e. Migrating registry from 1.23.2 to 2.0.0-M1 or 1.24.0 (RC3) works fine.
("Migrating" meaning a copying of the database and flow_storage directories
from the root directory of the older registry to 2.0.0-M1/1.24.0 (RC3))
Registry is having difficulty starting up. Has anyone else noticed this?
Caused by: org.h2.jdbc.JdbcSQLNonTransientConnectionException: Unsupported
database file version or invalid file header in file "//registry_2_0_0_M1/nifi-registry-2.0.0-M1/database/nifi-registry-primary.mv.db"
[90048-224] at
org.h2.message.DbException.getJdbcSQLException(DbException.java:690) at
org.h2.message.DbException.getJdbcSQLException(DbException.java:489) at
org.h2.message.DbException.get(DbException.java:212) at
org.h2.mvstore.db.Store.convertMVStoreException(Store.java:158) at
org.h2.mvstore.db.Store.(Store.java:142) at
org.h2.engine.Database.(Database.java:326) at
org.h2.engine.Engine.openSession(Engine.java:92) at
org.h2.engine.Engine.openSession(Engine.java:222) at
org.h2.engine.Engine.createSession(Engine.java:201) at
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:343)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:125) at
org.h2.Driver.connect(Driver.java:59) at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201) at
com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)
at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:100) at
com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
at
org.apache.nifi.registry.db.CustomFlywayConfiguration.getDatabaseType(CustomFlywayConfiguration.java:91)
... 121 common frames omittedCaused by:
org.h2.mvstore.MVStoreException: The write format 2 is smaller than the
supported format 3 [2.2.224/5] at
org.h2.mvstore.DataUtils.newMVStoreException(DataUtils.java:996) at
org.h2.mvstore.FileStore.getUnsupportedWriteFormatException(FileStore.java:943)
at
org.h2.mvstore.FileStore.processCommonHeaderAttributes(FileStore.java:547)
at
org.h2.mvstore.RandomAccessStore.readStoreHeader(RandomAccessStore.java:227)
at org.h2.mvstore.FileStore.start(FileStore.java:916) at
org.h2.mvstore.MVStore.(MVStore.java:289) at
org.h2.mvstore.MVStore$Builder.open(MVStore.java:2035) at
org.h2.mvstore.db.Store.(Store.java:133) ... 136 common frames
omitted
Thanks,
Nissim Shiman
On Tuesday, November 21, 2023 at 10:40:25 PM UTC, David Handermann
wrote:
Team,
I am pleased to be calling this vote for the source release of Apache
NiFi 2.0.0-M1.
Please review the following guide for how to verify a release candidate build:
https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
The source being voted on the and the convenience binaries are
available on the Apache Distribution Repository:
https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M1
The build artifacts are available on the Apache Nexus Repository:
https://repository.apache.org/content/repositories/orgapachenifi-1248
Git Tag: nifi-2.0.0-M1-RC5
Git Commit ID: f81a3597d40f3fe6df93d55347a11474ee6af2c8
GitHub Commit Link:
https://github.com/apache/nifi/commit/f81a3597d40f3fe6df93d55347a11474ee6af2c8
Checksums of nifi-2.0.0-M1-source-release.zip
SHA256: 217096b30c211b8fb6bd069832416b507b258f335d7301772a8c047f470a4b30
SHA512:
949849c59d2d8a2d0a441fd14c718f97cfb86c72fdd2de9a954a5829eebad6f9ad60622ecc6db200e246f77f771530be20f47793b3a8a18d2ceed252223139ee
Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/exceptionfactory.asc
KEYS file is available on the Apache Distribution Repository:
https://dist.apache.org/repos/dist/release/nifi/KEYS
Issues resolved for this version: 958
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12339599
Release note highlights can be found on the project wiki:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1
The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test.
Then please vote:
[] +1 Release this package as nifi-2.0.0-M1
[] +0 no opinion
[] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 1.24.0 (RC3)
+1 (non-binding) verified source release sha256/512 checksums successfully ran build using: Apache Maven 3.9.5 Java openjdk version: 1.8.0_382 linux kernel 3.10.0-1160 Ran various simple flows successfully. Migrated registry from 1.23.2 to this RC successfully. (i.e. and tested importing PGs, committing new versions of PGs to registry successfully.) Tested/verified: NIFI-11782 NPE when adding Snippet with label into Process Group NIFI-12084 Logging by process group is not workin on 1.x support branch (thank you to all those who worked on this and its parent ticket, NIFI-3065. This new granular level of logging is really nice) While testing, noticed/filed non holdup issues: NIFI-12387 Flow Configuration History can record a Comments change for Controller Service when no changes have been made NIFI-12388 Process Group log file suffix can become out of sync with its name when PG was copied from a version controlled PG Thank you Pierre and team for the upcoming release! Nissim Shiman On Friday, November 17, 2023 at 08:10:20 PM UTC, Dan S wrote: +1 (non-binding) - Went through the Release Candidate Verification <https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification> - Verified signatures and hashes - Built on CentOS Linux 7 - Java version Oracle 17.0.7 - Maven version 3.9.5 - Executed simple flow to exercise NIFI-11197 <https://issues.apache.org/jira/browse/NIFI-11197> - Confirmed fix for NIFI-12165 <https://issues.apache.org/jira/browse/NIFI-12165> - Confirmed fix for NIFI-11959 <https://issues.apache.org/jira/browse/NIFI-11959> Thanks for RM'ing Pierre! On Fri, Nov 17, 2023 at 1:01 PM Marton Szasz wrote: > +1 (binding) > > Compiled and ran with Java 17 on Ubuntu 22.04, executing a simple flow. > > On Fri, Nov 17, 2023 at 6:59 PM Matt Burgess wrote: > > > > +1 (binding) > > > > Ran through release helper, ran NiFi standalone with Java 8, tested > > various flows and components. Thanks for RM'ing Pierre! > > > > On Thu, Nov 16, 2023 at 2:01 PM Pierre Villard > > wrote: > > > > > > Team, > > > > > > I am pleased to be calling this vote for the source release of Apache > NiFi > > > 1.24.0. > > > > > > Please review the following guide for how to verify a release candidate > > > build: > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification > > > > > > The source being voted on the and the convenience binaries are > available on > > > the Apache Distribution Repository: > > > > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.24.0 > > > > > > The build artifacts are available on the Apache Nexus Repository: > > > > > > https://repository.apache.org/content/repositories/orgapachenifi-1241 > > > > > > Git Tag: nifi-1.24.0-RC3 > > > Git Commit ID: 586a1a8789e9c39914f4a6088ac26e22d60eeb91 > > > GitHub Commit Link: > > > > https://github.com/apache/nifi/commit/586a1a8789e9c39914f4a6088ac26e22d60eeb91 > > > > > > Checksums of nifi-1.24.0-source-release.zip > > > > > > SHA256: > 590cf9b1219f9fd66c81cc1740b2e245d85e341cdc280c769d354084dc27ee8a > > > SHA512: > > > > 8d3b9cb9c4686242620ad40ad83fadb972403ee70a101cbb6fa0116b54ad7793702da230871281c0de40322ddfdbfc89dacba7b690465e7b2329850dca5132e8 > > > > > > Release artifacts are signed with the following key: > > > > > > https://people.apache.org/keys/committer/pvillard.asc > > > > > > KEYS file is available on the Apache Distribution Repository: > > > > > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > > > Issues resolved for this version: 280 > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353443 > > > > > > Release note highlights can be found on the project wiki: > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.24.0 > > > > > > The vote will be open for 72 hours. > > > > > > Please download the release candidate and evaluate the necessary items > > > including checking hashes, signatures, build from source, and test. > Then > > > please vote: > > > > > > [] +1 Release this package as nifi-1.24.0 > > > [] +0 no opinion > > > [] -1 Do not release this package because... >
Re: [VOTE] Release Apache NiFi 1.23.0 (RC3)
+1 (non-binding) verified source release sha256/512 checksums successfully ran build using: Apache Maven 3.6.3 Java openjdk version: 1.8.0_382 linux kernel 3.10.0-1160 Ran various simple flows successfully. Wheil testing, noticed that gui managing Controller Services no longer automatically updates Last Updated time Created https://issues.apache.org/jira/browse/NIFI-11871 Also, noticed that many times, controller services go into validating state after being disabled, which only clears when controller service gui is manually refreshed Created https://issues.apache.org/jira/browse/NIFI-11872 Verified these issues where in existance in prior releases as well. Not blockers. Thank you David and team for the upcoming release! Nissim Shiman On Friday, July 28, 2023 at 03:57:25 PM UTC, Ferenc Kis wrote: +1 (non-binding) Went through the helper guide, full clean build with contrib check, verified signatures and hashes Maven home: /Users/fkis/.sdkman/candidates/maven/current Java version: 11.0.18, vendor: Eclipse Adoptium, runtime: /Users/fkis/.sdkman/candidates/java/11.0.18-tem Default locale: en_HU, platform encoding: UTF-8 OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac" Validation: - Started NiFi, created a simple flow - Started MiNiFi Java: created a simple flow and pushed to MiNiFi via C2 protocol Thanks for RMing, David! On Fri, Jul 28, 2023 at 3:31 PM Mark Bean wrote: > +1 (non-binding) > > Verified signature and hashes > Built source code on Ubuntu 20.04.2 using -Pcontrib-check and an empty > local repo each time > - openjdk version "1.8.0_362" > - openjdk version "11.0.19" > - openjdk version "17.0.7" > > Performed upgrade installation of both NiFi and NiFi Registry > > Verified NIFI-11845 (H2 database upgrade) > Ran some simple flows; no issues or problems found > > Thanks for RMing and timely assist on NIFI-11845 David. > > -Mark > > > On Thu, Jul 27, 2023 at 6:23 PM Peter Turcsanyi > wrote: > > > +1 (binding) > > > > Verified signatures and hashes. > > Built NiFi on Ubuntu 20.04 with Java 8 (Adoptium 1.8.0_382-b05), Java 11 > > (Adoptium 11.0.20+8) and Java 17 (Adoptium 17.0.8+7), Maven 3.9.3 with > > empty local repo. > > > > Ran flows for validating: > > - NIFI-11758 Add local file upload option in PutAzure*Storage processors > > - NIFI-11699 Fix dynamic properties in SnowflakeComputingConnectionPool > > - NIFI-11708 Upgrade snowflake-ingest-sdk to 2.0.1 and snowflake-jdbc to > > 3.13.33 > > > > Thanks for RMing David! > > > > Regards, > > Peter Turcsanyi > > > > On Thu, Jul 27, 2023 at 7:02 PM Nandor Soma Abonyi > > wrote: > > > > > +1 (binding) > > > > > > Went through the release helper guide. > > > Connected to NiFi registry, imported and committed some flows. > > > > > > Verified new features: > > > - NIFI-11758 Add local file upload option in PutAzure*Storage > processors > > > > > > Verified for regression: > > > - Azure ADLS and Blob_v12 related flows > > > - MQTT related flows > > > > > > Apache Maven 3.9.0 (9b58d2bad23a66be161c4664ef21ce219c2c8584) > > > Maven home: /usr/share/maven > > > Java version: 1.8.0_362, vendor: Temurin, runtime: > /opt/java/openjdk/jre > > > Default locale: en_US, platform encoding: UTF-8 > > > OS name: "linux", version: "6.3.12-orbstack-00210-ga4f4fae8883e", arch: > > > "aarch64", family: "unix" > > > > > > Thanks for RM'ing David! > > > > > > Regards, > > > Soma > > > > > > > On 2023. Jul 25., at 23:32, David Handermann < > > > exceptionfact...@apache.org> wrote: > > > > > > > > Team, > > > > > > > > I am pleased to be calling this vote for the source release of Apache > > > NiFi > > > > 1.23.0. > > > > > > > > Please review the following guide for how to verify a release > candidate > > > > build: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > > > > > The source being voted upon and the convenience binaries are > available > > on > > > > the Apache Distribution Repository: > > > > > > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.0/ > > > > > > > > The build artifacts are available on the Apache Nexu
dynamic properties not recovering from ghost component state
Hello apache nifi devs,
I have run into an issue that appears to require a different way of handling
ghost components than we are currently, and would like input before proceeding
further.
Specifically, this is related to
https://issues.apache.org/jira/browse/NIFI-11570
Besides the example in the ticket, a good case where this issue can be seen is
with the ExecuteScript processor with two dynamic properties, one sensitive and
one not. On recovery from ghost component, both properties will be sensitive.
The challenge is that on recovery from ghost component there no way of knowing
if a dynamic property is sensitive or not, so it is left in the existing
sensitive/encrypted state.
There are solutions, but each is a shift from how nifi is currently working.
1. Leave unencrypted properties in the clear when entering ghost component
mode. This would work, but we'd lose protections of why encryption was done in
the first place, so probably a non-starter.
2. When encrypting properties in ghost mode, have a custom encryption tag in
flow.json.gz/flow.xml.gz. i.e. instead of sensitive values starting with enc{
they would start with enc-ghost{ This will help when exiting ghost mode
where we could now more easily determine why a value was encrypted in the first
place, but is introducing new complexities in keeping track of why something is
encrypted
3. Leave handling of ghost components as is, but for a given component only
allow Dynamic Properties to be non-sensitive or senstive, but not both. This
would affect ExecuteScript, InvokeHTTP (and others) which currently allow both
and would be rolling back considerable work that was done to allow this, and
we'd lose existing functionality, so also not likely.
4. Leave as is. It is not so common to have components in ghost state, and
only some of them allow Dynamic Properties so this is an acceptable risk
considering the alternatives.
I am interested in community feedback and/or other suggestions to the current
situation.
Thanks,
Nissim Shiman
Re: [VOTE] Release Apache NiFi 1.22.0 (RC1)
+1 (non-binding) verified source release sha256/512 checksums successfully ran buildusing:Apache Maven 3.6.3 Java openjdk version: 11.0.17linux kernel 3.10.0-1160 VerifiedNIFI-11570 Bulletin causes UI to think it cannot refreshNIFI-11392 NiFi does not stop timely when using CRON schedulingNIFI-11109 registry client class name modified in flow.json/xml when missing nifi-flow-registry-client-nar Ran some simple flows successfully. Noticed that what appears to be temporary file is being created in nifi root. Created https://issues.apache.org/jira/browse/NIFI-11660This is not a blocker. Also, noticed that MonitorMemory has difficulty running when scheduled as a cron.Created https://issues.apache.org/jira/browse/NIFI-11661Also not a blocker. Thank you for the upcoming release! Nissim Shiman On Tuesday, June 6, 2023 at 09:38:43 PM UTC, Joe Witt wrote: Hello, I am pleased to be calling this vote for the source release of Apache NiFi 1.22.0. The source zip, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachenifi-1228 The source being voted upon and the convenience binaries can be found at: https://dist.apache.org/repos/dist/dev/nifi/nifi-1.22.0/ A helpful reminder on how the release candidate verification process works: https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate The Git tag is nifi-1.22.0-RC1 The Git commit ID is 71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8 https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8 https://github.com/apache/nifi/commit/71e3ea9f2c58d0cf2ce6824c388f2bd3e917dfc8 Checksums of nifi-1.22.0-source-release.zip: SHA256: d63a5f1db95160434670f112a0ee6e06fa141182bd0f12f0e58e3229991f3743 SHA512: 5f6da64e75a2d5446f1f274fe3de7e97290f5b916eabbc0491a99c6db33f74fbdea001b403044e75bc5c2183fb0500dfc143d0f4cba19d3511f866fff774d7f5 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/joewitt.asc KEYS file available here: https://dist.apache.org/repos/dist/release/nifi/KEYS 186 issues were closed/resolved for this release: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353069 Release note highlights can be found here: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.22.0 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [ ] +1 Release this package as nifi-1.22.0 [ ] +0 no opinion [ ] -1 Do not release this package because...
Re: [discuss] nifi 1.22 looking worthy at this point
I see NIFI-11392 has already been taken care of. Thank you Joe and markap14 On Tuesday, May 30, 2023 at 10:07:45 AM EDT, Joe Witt wrote: Thanks Michael and Nissim. Will flag both for follow up as part of 1.22. On Wed, May 24, 2023 at 12:35 PM Michael Moser wrote: > Team, > > I observed something wonky with the UI in 1.22.0-SNAPSHOT (using the latest > support/nifi-1.x branch) and I wrote NIFI-11560 [1] for it. I would be > sad if we release 1.22.0 and then decide this is significant, so any extra > eyes on it beforehand would be appreciated. > > [1] - https://issues.apache.org/jira/browse/NIFI-11560 > > Thanks, > -- Mike > > > On Wed, May 24, 2023 at 2:13 PM Nissim Shiman > wrote: > > > Joe and team, > > > > Thank you for the coming release. > > I am hoping someone can review/commit > > https://issues.apache.org/jira/browse/NIFI-11392for this as well. > > This resolves an issue where nifi gets hung on shutdown. > > I have reviewed and tested this and it looks good. > > > > (and thank you very much markap14 and mosermw for previous recent > > reviews/merging of tickets I was working on) > > Thanks, > > Nissim Shiman > > > > > > On Wednesday, May 24, 2023 at 12:57:42 PM EDT, David Handermann < > > exceptionfact...@apache.org> wrote: > > > > Joe, > > > > Thanks for initiating the discussion, I agree that the number of bug > fixes, > > dependency upgrades, and other improvements warrants a new 1.22.0 release > > version. This would also incorporate some additional deprecations, which > > will be helpful as we continue making progress on NiFi 2.0 goals. > > > > The system tests have had sporadic failures recently, so it seems best to > > wrap up NIFI-11591 [1] before proceeding. > > > > Regards, > > David Handermann > > > > [1] https://issues.apache.org/jira/browse/NIFI-11591 > > > > On Wed, May 24, 2023 at 11:39 AM Joe Witt wrote: > > > > > Team, > > > > > > > > > > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%201.22.0 > > > > > > It appears NiFi 1.22 has enough weight to be worth kicking out a > release. > > > 120+ JIRAs included several performance enhancers, bug fixes, a few new > > > features like ModifyCompression, PutGCP, etc.. > > > > > > Anything we're specifically waiting on? > > > > > > Thanks > > > > > >
Re: [discuss] nifi 1.22 looking worthy at this point
Joe and team, Thank you for the coming release. I am hoping someone can review/commit https://issues.apache.org/jira/browse/NIFI-11392for this as well. This resolves an issue where nifi gets hung on shutdown. I have reviewed and tested this and it looks good. (and thank you very much markap14 and mosermw for previous recent reviews/merging of tickets I was working on) Thanks, Nissim Shiman On Wednesday, May 24, 2023 at 12:57:42 PM EDT, David Handermann wrote: Joe, Thanks for initiating the discussion, I agree that the number of bug fixes, dependency upgrades, and other improvements warrants a new 1.22.0 release version. This would also incorporate some additional deprecations, which will be helpful as we continue making progress on NiFi 2.0 goals. The system tests have had sporadic failures recently, so it seems best to wrap up NIFI-11591 [1] before proceeding. Regards, David Handermann [1] https://issues.apache.org/jira/browse/NIFI-11591 On Wed, May 24, 2023 at 11:39 AM Joe Witt wrote: > Team, > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%201.22.0 > > It appears NiFi 1.22 has enough weight to be worth kicking out a release. > 120+ JIRAs included several performance enhancers, bug fixes, a few new > features like ModifyCompression, PutGCP, etc.. > > Anything we're specifically waiting on? > > Thanks >
request for ticket review
Hello apache nifi committers, Could someone review https://issues.apache.org/jira/browse/NIFI-10756 StandardSSLContextService will fail silently on nifi restart if keystore no longer exists I would very much appreciate it. Thanks, Nissim Shiman
Re: [VOTE] Release Apache NiFi 1.21.0 (RC2)
+1 (non-binding) verified source release sha256/512 checksums cleared local repo and successfully ran build using:Apache Maven 3.6.3 Java openjdk version: 11.0.17 linux kernel 3.10.0-1160 Verified NIFI-4718/NIFI-11166 Improve Detection of FlowFile V3 in IdentityMimeType Tested with flowfile-v3 containing video/x-ms-wmv file Ran some simple flows successfully. Ran into a question about connection backpressure settings (NIFI-11388) but that turned out to be something of a false positive. Found issue where links where Controller Services links that reference each other (in Controller Service Details's Settings tab) are sometimes not working. Made NIFI-11389 for this. Not a blocker. Thank you for upcoming release! Nissim Shiman On Tuesday, April 4, 2023 at 12:11:38 AM UTC, Joe Witt wrote: Hello, I am pleased to be calling this vote for the source release of Apache NiFi 1.21.0. The source zip, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachenifi-1226 The source being voted upon and the convenience binaries can be found at: https://dist.apache.org/repos/dist/dev/nifi/nifi-1.21.0/ A helpful reminder on how the release candidate verification process works: https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate The Git tag is nifi-1.21.0-RC2 The Git commit ID is 892f822107da84ca0dcfefdb5c91c5bc11dc3dc1 https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=892f822107da84ca0dcfefdb5c91c5bc11dc3dc1 Checksums of nifi-1.21.0-source-release.zip: SHA256: 598bf8e90f871f4ca25709dd4ecf3da16c814c08c0d8b2c8744dbd34df34dea5 SHA512: 58c10976bc22fb5406d9d1d9b7ca7d90c2dbed99565d00f1172bb33b375e9e068da5003d9dbbd87d2b17626e4e310b999c8581718532934e855c2134be763f04 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/joewitt.asc KEYS file available here: https://dist.apache.org/repos/dist/release/nifi/KEYS 126 issues were closed/resolved for this release: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12352899 Release note highlights can be found here: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.21.0 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [ ] +1 Release this package as nifi-1.21.0 [ ] +0 no opinion [ ] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 1.20.0 (RC1)
+1 (non-binding) verified source release sha256/512 checksums built and ran successfully using:Apache Maven 3.6.3 Java openjdk version: 1.8.0_352 linux kernel 3.10.0-1160 Verified various simple flows. Noticed issue where content viewer doesn't handle very large files (i.e. 100MB) well. nifi screen looses look and feel when this occurs and displays OutOfMemoryError Put in a jira for this https://issues.apache.org/jira/browse/NIFI-11153 This would be a very limited user initiated edge case, so I don't think this is a blocker. Thank you for the upcoming release! Nissim Shiman On Monday, February 6, 2023 at 08:56:10 PM UTC, Joe Witt wrote: Hello, I am pleased to be calling this vote for the source release of Apache NiFi 1.20.0. The source zip, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachenifi-1220 The source being voted upon and the convenience binaries can be found at: https://dist.apache.org/repos/dist/dev/nifi/nifi-1.20.0/ A helpful reminder on how the release candidate verification process works: https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate The Git tag is nifi-1.20.0-RC1 The Git commit ID is 81296b5b69a69d26afb8f8dec3a58a8363653890 https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=81296b5b69a69d26afb8f8dec3a58a8363653890 Checksums of nifi-1.20.0-source-release.zip: SHA256: 694e9eec951caf628576a2aaa16e7ddadc11b9bf455f16d503bdd88aefdbfe66 SHA512: f54891aadbf58eaf4df465e99eb935ddbb47c70c1e329968098762f670eeed56a427ed88f21f150c056f8b057f7120127f3470afe4f4f89b80d659d7b8080339 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/joewitt.asc KEYS file available here: https://dist.apache.org/repos/dist/release/nifi/KEYS 205 issues were closed/resolved for this release: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12352581 Release note highlights can be found here: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.20.0 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [ ] +1 Release this package as nifi-1.20.0 [ ] +0 no opinion [ ] -1 Do not release this package because...
Re: request for review, NIFI-10608
resending to list... I would greatly appreciate it if a committer could look at the PR for https://issues.apache.org/jira/browse/NIFI-10608 Also, if a committer could look at the PR for https://issues.apache.org/jira/browse/NIFI-10853 (this is just a few lines modifications in a property descriptor) I would greatly appreciate that as well. Thanks, Nissim Shiman On Thursday, December 8, 2022 at 01:40:40 PM EST, Nissim Shiman wrote: Hello apache devs, Could someone review/merge: https://issues.apache.org/jira/browse/NIFI-10608 Copied Processor Group no longer contains non-referenced Controller Services This has already been looked over by fellow contributor, markobean, and is waiting on final review/commit Thanks, Nissim Shiman
request for review, NIFI-10608
Hello apache devs, Could someone review/merge: https://issues.apache.org/jira/browse/NIFI-10608 Copied Processor Group no longer contains non-referenced Controller Services This has already been looked over by fellow contributor, markobean, and is waiting on final review/commit Thanks, Nissim Shiman
Re: [VOTE] Release Apache NiFi 1.19.0 (RC1)
+1 (non-binding) verified source release sha256/512 checksums built and ran successfully using:Apache Maven 3.6.3 Java openjdk version: 1.8.0_332 linux kernel 3.10.0-1160 Verified various simple flows. Modifed name of connection relationship within flow, as well as backpressure, size threshold, load balance strategy, prioritizers and copied PG it was contained in. Copied connection retained all modified settings. Issue with backward compatibility of flow.xml.gz was noticed where removing flow.json.gz and restarting nifi can lead to some WARN logs. Many times nifi will start up anyway, but if nifi was using a registry client then it will not startup. This was more of a kick the tires test and I don't think should be a blocker (as it is unlikely a nifi admin will decide to remove their flow.json.gz file), but for those who want to use an older flow.xml.gz (I guess from before nifi 1.16) this might be noticed. Will put in a jira for this edge case. Thank you for the upcoming release! Nissim Shiman On Wednesday, November 23, 2022 at 03:58:27 PM UTC, Joe Witt wrote: Hello, I am pleased to be calling this vote for the source release of Apache NiFi 1.19.0. The source zip, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachenifi-1216 The source being voted upon and the convenience binaries can be found at: https://dist.apache.org/repos/dist/dev/nifi/nifi-1.19.0/ A helpful reminder on how the release candidate verification process works: https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate The Git tag is nifi-1.19.0-RC1 The Git commit ID is ec87bf93add2f645d2ea426002e0c24db54614ff https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=ec87bf93add2f645d2ea426002e0c24db54614ff Checksums of nifi-1.19.0-source-release.zip: SHA256: 909c4fce81b305af955540f83115474d139c132e886847df7b76e94efd3d4641 SHA512: b3ba95984841140aa2a9a2ea471dd446d6fbd71dba320f723839e0938a7873f847e415a88b7c38e59907ad051c896949f0c7093c6d5e6b5db549de929ec32c6e Release artifacts are signed with the following key: https://people.apache.org/keys/committer/joewitt.asc KEYS file available here: https://dist.apache.org/repos/dist/release/nifi/KEYS 221 issues were closed/resolved for this release: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12352150 Release note highlights can be found here: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.19.0 The vote will be open for at least 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [ ] +1 Release this package as nifi-1.19.0 [ ] +0 no opinion [ ] -1 Do not release this package because...
Re: copied PG not containing controller service from the original?
Joe, Sorry for the previous brevity, but for example: create new PG -> right click -> Configure -> Controller Sevices -> Add any controller service (AvroReader for example) copy PG new PG -> right click -> Configure -> Controller Services -> (new controller service will be not listed/ AvroReader does not show up here) Thanks, Nissim On Tuesday, October 4, 2022 at 11:14:30 AM EDT, Joe Witt wrote: Nissim In the vote you wrote " Noticed that a copied processor group will not contain controller service that original process group had if controller service had not been referenced to a processorFurther investigation shows this has been the case since at least apache nifi 1.16.3, so not a holdup" Can you tell us more about this in terms of what type of controller services you have? Thanks
Re: [VOTE] Release Apache NiFi 1.18.0 (RC4)
+1 (non-binding) verified source release sha256/512 checksums built and ran successfully using:Apache Maven 3.6.3 Java openjdk version: 1.8.0_332 linux kernel 3.10.0-1160 verified out of the box simple invokeHTTP to ListenHTTP flow put simple invokeHTTP to ListenHTTP flow in process group, verified flow modified InvokeHTTP name, bulletin level and comments and copied process group and verified flow continues to work as well as modified InvokeHTTP name, bulletin level and comments persisted in new PG Noticed that a copied processor group will not contain controller service that original process group had if controller service had not been referenced to a processorFurther investigation shows this has been the case since at least apache nifi 1.16.3, so not a holdup Thank you for the upcoming release! Nissim On Monday, October 3, 2022 at 08:45:32 PM UTC, Joe Witt wrote: Hello, I am pleased to be calling this vote for the source release of Apache NiFi 1.18.0. The source zip, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachenifi-1214 The source being voted upon and the convenience binaries can be found at: https://dist.apache.org/repos/dist/dev/nifi/nifi-1.18.0/ A helpful reminder on how the release candidate verification process works: https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate The Git tag is nifi-1.18.0-RC4 The Git commit ID is 109e54cd585902a981d1b370b3dc4d1620be438c https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=109e54cd585902a981d1b370b3dc4d1620be438c Checksums of nifi-1.18.0-source-release.zip: SHA256: 925cbb92c107d0fa3194a349d985cff4933a61b2555eff57b1b81433fe37c139 SHA512: f143215b1746342e7584f5ad65b546fcc378cd78aa17628fb605dfdbbaf11e897a0173dd67807fc90cb18c17124a4227d5fe07e7ed609d9ed1904503b757c604 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/joewitt.asc KEYS file available here: https://dist.apache.org/repos/dist/release/nifi/KEYS 184 issues were closed/resolved for this release: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12352150 Release note highlights can be found here: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.18.0 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [ ] +1 Release this package as nifi-1.18.0 [ ] +0 no opinion [ ] -1 Do not release this package because...
Re: request for ticket review
Hello apache nifi devs, Resending request ... in hopes of still making 1.18 Thanks, Nissim On Monday, September 19, 2022 at 03:52:49 PM EDT, Nissim Shiman wrote: Hello apache nifi committers, Could you look at two bug fixes: https://issues.apache.org/jira/browse/NIFI-9878 DistributedCacheMap Handshake failure, processor hang indefinitely and https://issues.apache.org/jira/browse/NIFI-10287 ExecuteScript processor not supporting Module Directory for python These have relatively modest amount of code changes and have already been reviewed. Thanks, Nissim Shiman
request for ticket review
Hello apache nifi committers, Could you look at two bug fixes: https://issues.apache.org/jira/browse/NIFI-9878 DistributedCacheMap Handshake failure, processor hang indefinitely and https://issues.apache.org/jira/browse/NIFI-10287 ExecuteScript processor not supporting Module Directory for python These have relatively modest amount of code changes and have already been reviewed. Thanks, Nissim Shiman
Load Distribution Service nar location
Hello nifi devs, Are we aware of a location for an implementation of the Load Distribution Service api [1] ? This is used in the DistributeLoad processor [2], but is not found when chosen. I searched the nifi code for an implementation and there doesn't appear to be one. Maybe it is in a supporting repository? To reproduce issue: set property "distribution strategy" to "load distribution service" -> Apply -> close processor -> re-enter processor (i.e. "Configure") -> "Load Distribution Service ID" -> service can not be created - error is "No controller service types found that are applicable for this property" [1] https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-standard-services/nifi-load-distribution-service-api/src/main/java/org/apache/nifi/loading/LoadDistributionService.java [2] https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java#L137 Thanks, Nissim
Re: request to commit reviewed PR - adding bulletin level control to Controller Services
Hello Apache NiFi team, Resending below email with request to review/commit PR: https://github.com/apache/nifi/pull/6035 It already has a review with +1, but is waiting on a committer to look at it. Thanks, Nissim On Thursday, May 19, 2022, 02:55:40 PM EDT, Nissim Shiman wrote: Hello Apache Nifi Committers, Could someone commit https://issues.apache.org/jira/browse/NIFI-9440 Allow Bulletin level to be configurable for Controller Services PR: https://github.com/apache/nifi/pull/6035 At the moment Controller Services have bulletins at the WARN level. This will allow users to have finer control over bulletin levels (i.e. settings of NONE, DEBUG, INFO, WARN, ERROR) just like processors currently allow. I think this functionality will be useful for many nifi users. This has already been reviewed. Thanks, Nissim Shiman
request to commit reviewed PR - adding bulletin level control to Controller Services
Hello Apache Nifi Committers, Could someone commit https://issues.apache.org/jira/browse/NIFI-9440 Allow Bulletin level to be configurable for Controller Services PR: https://github.com/apache/nifi/pull/6035 At the moment Controller Services have bulletins at the WARN level. This will allow users to have finer control over bulletin levels (i.e. settings of NONE, DEBUG, INFO, WARN, ERROR) just like processors currently allow. I think this functionality will be useful for many nifi users. This has already been reviewed. Thanks, Nissim Shiman
turning off bulletins for controller services
Hello NiFi devs, Do we know if there is way to turn off bulletins for controller services? Processors have Bulletin level configurable on the "Settings" tabs, but it is not clear there is similar functionality exposed for controller services Thanks, Nissim
"Active Threads" different on status bar then "Active Thread Count" when listed by nodes
Hello NiFi devs, "Active Threads" on the main gui's status bar has a lower number then "Active Thread Count" when broken down by nodes (i.e. 3 lines icon -> Cluster) It appears that "Active Threads" only captures threads by processors, and "Active Thread Count" is more comprehensive and includes has other threads as well (i.e. reporting tasks etc.) Is this correct? And do we know the various other sources that nifi threads can come from besides processors that "Active Thread Count" captures? Thanks, Nissim Shiman
ListenHTTP - filters http headers but not flowfile attributes
Hello Nifi devs, Currently ListenHTTP has the filtering property: "HTTP Headers to receive as Attributes (Regex)" This works except for the case where the data is from PostHTTP and was sent as a flowfile.In this scenario the flowfile attributes are not sent as headers, rather the flowfile contents and attributes are packaged together (as FFv3) and sent to ListenHTTP and there is no option for the ListenHTTP processor to do any filtering. What do we think about renaming the filtering property to: "Attributes to Receive (Regex)" and have it applied to HTTP headers (when dealing with non-FFv3 data) as well as incoming attributes (when dealing with FFv3 data) Thank You, Nissim Shiman
Re: [DISCUSS] Release of Apache NiFi 1.13.0
NIFI-7738 https://issues.apache.org/jira/browse/NIFI-7738 (Provenance Search Events - add feature to allow inverse query) has had PR updated based on comments. It would be very much appreciated if this could make it into 1.13.0 as well Thanks, Nissim On Tuesday, January 12, 2021, 01:58:30 PM EST, Matt Burgess wrote: I'm testing some final updates for NIFI-7989, will push up the PR this afternoon and Peter said he'd continue the review. On Tue, Jan 12, 2021 at 1:18 PM Pierre Villard wrote: > > Hi all, > > Current status about what we want to get in based on this thread: > - NIFI-8034 - PropertyValue.isExpressionLanguagePresent always returns true > for non-null values - https://github.com/apache/nifi/pull/4746 > - NIFI-7989 - Add Hive "data drift" processor - > https://github.com/apache/nifi/pull/4750 > - NIFI-7356 - TLS + embedded Zookeeper - pull request to be submitted by > Nathan > > If we can wrap up these JIRAs, we could get a release candidate for NiFi > 1.13 soon. > > Thanks, > Pierre > > > > Le mer. 6 janv. 2021 à 04:17, Joey Frazee > a écrit : > > > I have https://github.com/apache/nifi/pull/4632 which fixes an OOME in > > PutAzureBlobStorage reported in > > https://lists.apache.org/x/thread.html/rdef82be24828277b85bdc94dc57a8fb9df6f73552daeda289c941a51%40%3Cusers.nifi.apache.org%3E > > > > It’s a pretty small change. > > > > -joey > > > > > On Jan 5, 2021, at 3:14 PM, Matt Burgess wrote: > > > > > > I am reviewing a PR at the moment but intend to review the graph stuff > > right afterwards. > > > > > >> On Jan 5, 2021, at 5:35 PM, Mike Thomsen > > wrote: > > >> > > >> I have a PR for graph that we need to close out because part of the > > graph > > >> bundle will be broken without it. > > >> > > On Tue, Jan 5, 2021, 11:50 Mark Bean wrote: > > >>> > > >>> I'd like to see this PR completed as well > > >>> https://github.com/apache/nifi/pull/4225 > > >>> > > >>> There's been discussion on it, and I don't see any further objections. > > >>> > > >>> Thanks, > > >>> Mark > > >>> > > >>> On Thu, Nov 26, 2020 at 4:55 AM Pierre Villard < > > >>> pierre.villard...@gmail.com> > > >>> wrote: > > >>> > > Hi community, > > > > Starting a discussion thread around the release of NiFi 1.13.0. We > > added > > quite a lot of significant improvements and features since the > > release of > > 1.12.1 - I see 143 JIRAs with 1.13.0 as the fix version. I think it > > makes > > sense to consider a new release. > > > > Please share here the JIRAs with opened pull requests that we would > > need > > >>> to > > look at to make this release happen. > > > > Thanks, > > Pierre > > > > >>> > >
Re: request to merge PRs to main
Hello Nifi Devs, Could someone move https://issues.apache.org/jira/browse/NIFI-6242 into the coming 1.13.0 release. This is a bug fix I have reviewed and tested so it is ready for a merge. It handles a case where the SEND provenance event sometimes has the incorrect value for its transitUri. Also, thank you Mark Payne for merging NIFI-7225 and commenting on NIFI-7738! Thank You, Nissim Shiman On Monday, December 28, 2020, 12:25:25 PM EST, Nissim Shiman wrote: Hello apache nifi devs, I have thoroughly reviewed and tested PRs for 2 bug related tickets (both on java 8 and 11): https://issues.apache.org/jira/browse/NIFI-7225 and https://issues.apache.org/jira/browse/NIFI-6242 Also, I have worked a ticket https://issues.apache.org/jira/browse/NIFI-7738 that has been reviewed and tested by another contributer as well. So no additional reviewing/testing needs to be done by a PMC/committer as they have been vetted. Could someone commit these to main? Thank You! Nissim Shiman
request to merge PRs to main
Hello apache nifi devs, I have thoroughly reviewed and tested PRs for 2 bug related tickets (both on java 8 and 11): https://issues.apache.org/jira/browse/NIFI-7225 and https://issues.apache.org/jira/browse/NIFI-6242 Also, I have worked a ticket https://issues.apache.org/jira/browse/NIFI-7738 that has been reviewed and tested by another contributer as well. So no additional reviewing/testing needs to be done by a PMC/committer as they have been vetted. Could someone commit these to main? Thank You! Nissim Shiman
Re: [DISCUSS] Release of Apache NiFi 1.13.0
Hello Nifi Team, NIFI-7738 [1] https://github.com/apache/nifi/pull/4563 has already been reviewed/tested by a fellow contributer I would greatly appreciate a committer moving this to main for 1.13.0 Thanks, Nissim Shiman [1] https://issues.apache.org/jira/browse/NIFI-7738 On Sunday, November 29, 2020, 07:12:12 PM EST, Matt Burgess wrote: I just merged Mike's Cassandra DMC PR, reviewing the graph stuff now. I have a bunch of open PRs [1], I don't think any are blockers for 1.13.0 but thought I'd take this opportunity to find some reviewers :) Regards, Matt [1] https://github.com/apache/nifi/pulls/mattyb149 On Sat, Nov 28, 2020 at 10:13 AM Phillip Grenier wrote: > > I wouldn't mind https://github.com/apache/nifi/pull/4554 making it in. Have > seen a couple more instances where people have created custom processors to > work around this PR. > > On Thu, Nov 26, 2020 at 11:10 AM Mike Thomsen > wrote: > > > Also, for most of us (committers and community members), it's a > > holiday season so might be good to delay a release to early January > > anyway. Just a thought. > > > > On Thu, Nov 26, 2020 at 11:08 AM Mike Thomsen > > wrote: > > > > > > https://github.com/apache/nifi/pull/4638 - This is a big increase in > > > graph functionality, and I want to give him a chance to get his first > > > contribution in with 1.13 (we may have some more of his work we can > > > open source). > > > > > > Matt's also reviewing some of my Cassandra and graph tickets as well. > > > I wouldn't call those **blockers**, but having the Cassandra DMC fully > > > fleshed out would be nice. > > > > > > On Thu, Nov 26, 2020 at 9:10 AM Otto Fowler > > wrote: > > > > > > > > NIFI-7795 https://github.com/apache/nifi/pull/4656 > > > > NIFI-7761 https://github.com/apache/nifi/pull/4513 > > > > NIFI-2072 https://github.com/apache/nifi/pull/4384 > > > > > > > > From: Pierre Villard > > > > > > > > Reply: dev@nifi.apache.org > > > > Date: November 26, 2020 at 04:55:56 > > > > To: dev@nifi.apache.org > > > > Subject: [DISCUSS] Release of Apache NiFi 1.13.0 > > > > > > > > Hi community, > > > > > > > > Starting a discussion thread around the release of NiFi 1.13.0. We > > added > > > > quite a lot of significant improvements and features since the release > > of > > > > 1.12.1 - I see 143 JIRAs with 1.13.0 as the fix version. I think it > > makes > > > > sense to consider a new release. > > > > > > > > Please share here the JIRAs with opened pull requests that we would > > need to > > > > look at to make this release happen. > > > > > > > > Thanks, > > > > Pierre > >
Re: FILE_EXISTS_VALIDATOR allows directories as well, is this a feature or a bug?
Thank you Mike and Lars for your feedback! I like your idea of "a second constructor that allows you to set a boolean flag to put in a "file-only" mode."I think that is the most graceful way forward. I think we are OK with not having another validator for validating folders as being folders as it seems that it can be done using the code that already exists [1][2] [1] https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java#L115[2] https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-reporting-tasks/src/main/java/org/apache/nifi/controller/MonitorDiskUsage.java#L55 Thank You! Nissim On Saturday, November 28, 2020, 03:56:38 PM EST, Mike Thomsen wrote: Lars, I think we'd be better off keeping it simple like FILE_EXISTENCE_VALIDATOR and FOLDER_EXISTENCE_VALIDATOR. I'm not sure that symlinks are really an issue here for validation purposes. Regardless, we'd need to give community members with their own custom bundles a good 2-3 releases worth of @Deprecated warning on the current validator before removing it would be something we could consider. Thanks, Mike On Sat, Nov 28, 2020 at 1:36 PM Lars Winderling wrote: > > Mike, > > what about giving it another name then like REAL_FILE_VALIDATOR? It's not > perfect, but it draws from the distinction between real (proper) files on the > one, and symlinks, dirs etc in the other side, like many apis use it. I'm not > quite sure, what do you think? > Best, > Lars > > On 28 November 2020 16:37:41 CET, Mike Thomsen wrote: > >Nissim, > > > >If I had to guess, it was an oversight that it handles both folders > >and files. Any refactored version would have to not change the current > >default behavior. So, what I'd suggest would be a second constructor > >that allows you to set a boolean flag to put in a "file-only" mode and > >then add another validator that checks for the existence of a folder > >and that it is a folder, not a file. > > > >Thanks, > > > >Mike > > > >On Thu, Nov 19, 2020 at 1:06 PM Nissim Shiman > > wrote: > >> > >> Hello Nifi devs, > >> It has been noticed that the StandardValidators.FILE_EXISTS_VALIDATOR > >[1] verifies that a value exists, but it does not verify that it is a > >file, as opposed to being a directory [2] > >> > >> Is this by design or is it worthy of a bug fix? > >> > >> > >> [1] > >https://github.com/apache/nifi/blob/main/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/processor/util/StandardValidators.java#L505[2] > >https://github.com/apache/nifi/blob/main/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/processor/util/StandardValidators.java#L851-L852 > >> > >> Thanks, > >> Nissim Shiman
FILE_EXISTS_VALIDATOR allows directories as well, is this a feature or a bug?
Hello Nifi devs, It has been noticed that the StandardValidators.FILE_EXISTS_VALIDATOR [1] verifies that a value exists, but it does not verify that it is a file, as opposed to being a directory [2] Is this by design or is it worthy of a bug fix? [1] https://github.com/apache/nifi/blob/main/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/processor/util/StandardValidators.java#L505[2] https://github.com/apache/nifi/blob/main/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/processor/util/StandardValidators.java#L851-L852 Thanks, Nissim Shiman
Re: request to review NIFI-7738, Provenance Search Events - add reverse query capability
Hello Joe, Phillip and Pierre, I am really happy we are having this discussion. I (and very likely other contributers) was under the impression that only reviews by committers/PMC members (i.e. more senior nifi dev community members) had any real weight, so that is why I have not been reviewing standing PRs. So this discussion is bringing out out that anyone can review/test open PRs and once any issues in the PRs have been smoothed out that it shouldn't be much of an issue to have it committed at that point. Actually, it looks like a fellow contributer, Mark Bean, has already noticed this conversation and has already reviewed my PR and has made some useful comments. So we are already seeing results. (and thank you Mark!) I plan to start reviewing/testing PRs. Thank You, Nissim Shiman On Thursday, October 8, 2020, 12:17:13 PM EDT, Joe Witt wrote: Hello Anyone can do a review of the code and build and feasibility of a change. It only requires commit privileges to actually merge something. Doing the hard work of reviewing/testing the PRs is a *great* way to earn merit. Much of the PRs that tend to hang around are from people contributing a specific thing and those are some of the most challenging. We as a community then take on support of those things for function, License and Notice adherence, etc.. On the other hand, these folks are also the ones that can often start contributing elsewhere and building merit and earning commit and even PMC status. So we have to invest one way or another. But for sure this is not easy to do well as time has shown. Nissim, in your case the contribution is probably cool/good/useful. What could help you get more attention on it is helping with bug fixes or reviewing other peoples contributions/etc.. Not saying anyone is checking whether you do that or not but if they see more of your effort it *does* help improve likelihood of getting merge attention. Not required...just sharing another perspective. Regardless thanks for the contribution and hopefully someone is able to review soon. Thanks On Thu, Oct 8, 2020 at 9:00 AM Phillip Grenier wrote: > Pierre, > > I think this discussion brings up a valid conversation point. At some point > a PMC member needs to approve the merge request, so from a contributors > level what can we do to make that merge both easier and/or more likely to > happen. That and how the community can help filter down the ever growing > backlog of PRs. > > Thanks and look forward to helping, > > Phillip > > On Wed, Oct 7, 2020 at 2:04 PM Pierre Villard > > wrote: > > > Nissim, > > > > Thanks a lot for your contribution but there are 277 pull requests opened > > at this time. This is a community effort, and if you expect people to > > review your PRs, you'd have to also try reviewing PRs opened by others in > > the community. Otherwise this will need to wait for someone with some > > bandwidth to focus on this. > > > > Thanks, > > Pierre > > > > Le mer. 7 oct. 2020 à 19:59, Nissim Shiman a > > écrit : > > > > > Second attempt... > > > > > > > > > Hello NiFi team, > > > > > > Could someone respond to this email. > > > > > > Thanks, > > > Nissim Shiman > > > On Monday, October 5, 2020, 11:01:57 AM EDT, Nissim Shiman > > > wrote: > > > > > > Hello NiFi team, > > > Could someone review the pull request for NIFI-7738 ( > > > https://issues.apache.org/jira/browse/NIFI-7738), that was put in 5 > days > > > ago? > > > (Provenance Search Events - add feature to allow inverse query) > > > > > > One case this will come in handy is if a particular flowfile has an > issue > > > and goes in circles through processors and ends up emitting a lot of > > > provenance making it hard to follow other flowfiles. > > > > > > This feature will allow users to query for all provenance except for > the > > > uuid of the problematic flowfile. > > > > > > > > > > > > This can also be used for any indexed field or attribute as well. > > > This has been compiled and tested on java 8 and java11 for all 4 > > > provenance repository implementations for both indexed fields and > > indexed > > > attributes > > > The only "real" code changes were made to LuceneUtil.java (to handle > > > WriteAhead, EncryptedWriteAhead and Persistent Provenance Repostories) > > and > > > VolatileProvenanceRepository.java > > > The rest are just involved in propagating the ability to query the > "NOT" > > > option for a given user entered provenance search value from the gui to > > the > > > backend query. > > > > > > Thank You, > > > Nissim Shiman > > >
Re: request to review NIFI-7738, Provenance Search Events - add reverse query capability
Second attempt... Hello NiFi team, Could someone respond to this email. Thanks, Nissim Shiman On Monday, October 5, 2020, 11:01:57 AM EDT, Nissim Shiman wrote: Hello NiFi team, Could someone review the pull request for NIFI-7738 (https://issues.apache.org/jira/browse/NIFI-7738), that was put in 5 days ago? (Provenance Search Events - add feature to allow inverse query) One case this will come in handy is if a particular flowfile has an issue and goes in circles through processors and ends up emitting a lot of provenance making it hard to follow other flowfiles. This feature will allow users to query for all provenance except for the uuid of the problematic flowfile. This can also be used for any indexed field or attribute as well. This has been compiled and tested on java 8 and java11 for all 4 provenance repository implementations for both indexed fields and indexed attributes The only "real" code changes were made to LuceneUtil.java (to handle WriteAhead, EncryptedWriteAhead and Persistent Provenance Repostories) and VolatileProvenanceRepository.java The rest are just involved in propagating the ability to query the "NOT" option for a given user entered provenance search value from the gui to the backend query. Thank You, Nissim Shiman
request to review NIFI-7738, Provenance Search Events - add reverse query capability
Hello NiFi team, Could someone review the pull request for NIFI-7738 (https://issues.apache.org/jira/browse/NIFI-7738), that was put in 5 days ago? (Provenance Search Events - add feature to allow inverse query) One case this will come in handy is if a particular flowfile has an issue and goes in circles through processors and ends up emitting a lot of provenance making it hard to follow other flowfiles. This feature will allow users to query for all provenance except for the uuid of the problematic flowfile. This can also be used for any indexed field or attribute as well. This has been compiled and tested on java 8 and java11 for all 4 provenance repository implementations for both indexed fields and indexed attributes The only "real" code changes were made to LuceneUtil.java (to handle WriteAhead, EncryptedWriteAhead and Persistent Provenance Repostories) and VolatileProvenanceRepository.java The rest are just involved in propagating the ability to query the "NOT" option for a given user entered provenance search value from the gui to the backend query. Thank You, Nissim Shiman
potential new feature - inverse Provenance queries
Hello NiFi community, I believe a feature that NiFi administrators would find useful is doing an inverse search on provenance. So for example, instead of just being able to search on provenance to find provenance with Event Type of SEND we also have the option of finding all provenance that does NOT have Event Type of SEND. The underlying apache lucene already supports Boolean queries. Actually, NiFi is already doing a Boolean query. But now instead of doing an Occur.MUST, NiFi will also be able to do an Occur.MUST_NOT [1] I am thinking there will be a GUI change as well, as there will be a "NOT" or "INVERSE" column header at the far right of the Provenance's Search Events pop up, with checkboxes underneath it (i.e. a checkbox to the right of each fieldname/value that a user can query provenance events for. If the checkbox is checked, then the user is indicating to do an inverse query for the value entered.) I put together a proof of concept and it works without any obvious complexity or gotcha issues. What do you think about this? Thank You, Nissim Shiman [1] https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/lucene/LuceneUtil.java#L108
can contributers review PRs?
Hello NiFi Devs, It seem like PRs are reviewed by committers. Can a contributor review a PR as well? And, if so, could that count as the necessary review for the ticket? Some tickets have nice functionality, but could become stale if committers have their bandwidth pulled in other directions. This way the final merge is still done by a committer, but the review step has already been handled. Thanks, Nissim SHiman
onPropertyModified() is triggered on Processors and Controller Services, even when properties are the defaults, bug or feature?
Hello Nifi Devs, Every Processor and ControllerService inherits from ConfigurableComponent [1] and therefore has onPropertyModified() [2] as well. Currently, this method is run at nifi startup on each property with a value, even if the value is the default. (i.e. and it has never been anything other than the default value). Is this a bug or a feature? [1] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/components/ConfigurableComponent.java [2] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/components/ConfigurableComponent.java#L68 Thanks, Nissim Shiman
Re: [discuss] Apache NiFi 1.11.0
Could someone look at merging in NIFI-6851 https://issues.apache.org/jira/browse/NIFI-6851 Thanks, Nissim Shiman On Monday, January 6, 2020, 12:06:06 PM EST, Joe Witt wrote: Team, We've already seen a good stream of nice fixes, improvements, and some minor features land on NIFi 1.11.0. It is enough to my eyes to warrant doing a release to pull these in plus it helps get back our release cadence mojo. I'll volunteer to RM this one as well as it almost makes me feel like a software developer still. Here are things already resolved for 1.11 https://issues.apache.org/jira/browse/NIFI-6956?jql=project%20%3D%20NIFI%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%201.11.0 Here are things open but tagged to 1.11 https://issues.apache.org/jira/browse/NIFI-6982?jql=project%20%3D%20NIFI%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20fixVersion%20%3D%201.11.0 I'l start cleaning/moving as necessary. Please advise if there are major things not present that we need to consider. But otherwise lets keep the ball moving forward and get after more regular releases again. Thanks Joe
jira contributer access request
Hello Apache Nifi team, I would like to request jira contributer access to apache nifi. I would like to work ticket NIFI-6851. My jira username is:Nissim Shiman Thanks, Nissim Shiman
Re: PULL ProvenanceEvent
Joe, Very nice... Thanks! Nissim On Wednesday, November 6, 2019, 1:05:09 PM EST, Joe Witt wrote: Nissim Notionally I am saying that session.getProvenanceReporter().receive(...) should have an option to call session.getProvenanceReporter().receive(...,ACTIVE|PASSIVE) and if not specified it would be UNSPECIFIED. I dont think this needs to be on the flowfile attribute - it would go straight to the provenance event itself which is generated by the session. Thanks Joe On Wed, Nov 6, 2019 at 11:32 AM Nissim Shiman wrote: > Joe, > > Just to verify what you mean, > > You are saying that the line: > flowfile = session.putAttribute(flowfile, "receiveType", "active") > > could be added before > session.getProvenanceReporter().receive(...) > > > to indicate a PULL. Is this correct? > > Thanks, > > Nissim > > > > > > > On Monday, November 4, 2019, 12:50:11 PM EST, Nissim Shiman > wrote: > > Having an attribute added indicating passive/active/query for RECEIVE > and FETCH will work, > > but nifi attributes are stateful (i.e. they will still be on the flowfile > as metadata a couple of processor steps down the flow) > > Maybe an option is to expand the the api for RECEIVE and FETCH for with a > new parameter for passive/active/query ? > (i.e. the existing message signatures, such as [1] will remain the same, > but new ones will be added to handle this new parameter? > > [1] > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > On Thursday, October 31, 2019, 10:10:40 PM EDT, Joe Witt < > joe.w...@gmail.com> wrote: > > These distinctions may be meaningful. Adding them as an attribute lets > the > meaning convey but not introduce complexity for the majority case which is > the distinction isnt key. > > thanks > > On Thu, Oct 31, 2019 at 4:05 PM Nissim Shiman > wrote: > > > Mike, > > I like the QUERY type as well. Basically a more refined PULL. Very > nice. > > > > > > Part of the challenge of adding PULL as a type is that there are > currently > > two flavors of RECEIVEs. > > RECEIVE and FETCH [1] > > > > So any addition of a PULL would need a second flavor of PULL to match the > > case where a flowfile's contents are being overwritten as well (i.e. as > > FETCH is currently doing) > > > > > > [1] > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java#L42 > > > > > > Thanks, > > Nissim > > > > > > On Wednesday, October 30, 2019, 6:41:04 PM EDT, Mike Thomsen < > > mikerthom...@gmail.com> wrote: > > > > I like the idea of creating PULL as a type. In fact, I'd propose that > > there > > are three scenarios here: > > > > RECEIVE - Passively acquire in a sort of hand-off situation. Ex: Kafka > > subscription > > PULL - Direct operations to seek out and fetch something in a targeted > > fashion. Ex. GetHttp > > QUERY - Go looking for the data and take what matches your search. Ex. > > JsonQueryElasticsearch, GetMongo, any SQL query processor, etc. > > > > > > > > On Wed, Oct 30, 2019 at 1:31 PM Nissim Shiman > > > wrote: > > > > > Joe, > > > > > > > > > It is hard to say how much value transit URI would bring to clarify a > > > RECEIVE. > > > For example a RECEIVE with transit URI of https: could be either > a > > > GetHTTP (i.e. active) or ListenHTTP (i.e. passive) > > > > > > but your idea of "a metadata item specifying active vs passive" is a > very > > > clever way to make this work with mimimal disruptions. > > > > > > My understanding of this is that the current receive() calls in > > > ProvenanceReporter [1] will remain the same, but news ones will be > added > > > with a boolean parameter reflecting if the receive is active or > passive. > > > This will allow the current list of Provenance Events [2] to remain the > > > same. So third party/custom processors can continue working as is > > > > > > Does this sound like what you are thinking? > > > > > > > > > [1] > > > > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > > > > [2] > > > apache/nifi > > > > > > > > > Thanks, > > > > > > Nissim > > > On
Re: PULL ProvenanceEvent
Joe, Just to verify what you mean, You are saying that the line: flowfile = session.putAttribute(flowfile, "receiveType", "active") could be added before session.getProvenanceReporter().receive(...) to indicate a PULL. Is this correct? Thanks, Nissim On Monday, November 4, 2019, 12:50:11 PM EST, Nissim Shiman wrote: Having an attribute added indicating passive/active/query for RECEIVE and FETCH will work, but nifi attributes are stateful (i.e. they will still be on the flowfile as metadata a couple of processor steps down the flow) Maybe an option is to expand the the api for RECEIVE and FETCH for with a new parameter for passive/active/query ? (i.e. the existing message signatures, such as [1] will remain the same, but new ones will be added to handle this new parameter? [1] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 On Thursday, October 31, 2019, 10:10:40 PM EDT, Joe Witt wrote: These distinctions may be meaningful. Adding them as an attribute lets the meaning convey but not introduce complexity for the majority case which is the distinction isnt key. thanks On Thu, Oct 31, 2019 at 4:05 PM Nissim Shiman wrote: > Mike, > I like the QUERY type as well. Basically a more refined PULL. Very nice. > > > Part of the challenge of adding PULL as a type is that there are currently > two flavors of RECEIVEs. > RECEIVE and FETCH [1] > > So any addition of a PULL would need a second flavor of PULL to match the > case where a flowfile's contents are being overwritten as well (i.e. as > FETCH is currently doing) > > > [1] > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java#L42 > > > Thanks, > Nissim > > > On Wednesday, October 30, 2019, 6:41:04 PM EDT, Mike Thomsen < > mikerthom...@gmail.com> wrote: > > I like the idea of creating PULL as a type. In fact, I'd propose that > there > are three scenarios here: > > RECEIVE - Passively acquire in a sort of hand-off situation. Ex: Kafka > subscription > PULL - Direct operations to seek out and fetch something in a targeted > fashion. Ex. GetHttp > QUERY - Go looking for the data and take what matches your search. Ex. > JsonQueryElasticsearch, GetMongo, any SQL query processor, etc. > > > > On Wed, Oct 30, 2019 at 1:31 PM Nissim Shiman > wrote: > > > Joe, > > > > > > It is hard to say how much value transit URI would bring to clarify a > > RECEIVE. > > For example a RECEIVE with transit URI of https: could be either a > > GetHTTP (i.e. active) or ListenHTTP (i.e. passive) > > > > but your idea of "a metadata item specifying active vs passive" is a very > > clever way to make this work with mimimal disruptions. > > > > My understanding of this is that the current receive() calls in > > ProvenanceReporter [1] will remain the same, but news ones will be added > > with a boolean parameter reflecting if the receive is active or passive. > > This will allow the current list of Provenance Events [2] to remain the > > same. So third party/custom processors can continue working as is > > > > Does this sound like what you are thinking? > > > > > > [1] > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > > [2] > > apache/nifi > > > > > > Thanks, > > > > Nissim > > On Tuesday, October 29, 2019, 12:47:40 PM EDT, Joe Witt < > > joe.w...@gmail.com> wrote: > > > > Nissim > > > > I like the idea to introduce a more refined type of event for how data is > > brought into nifi (active - PULL, passive - RECEIVE). > > > > That said it might be sufficient to simply have this distinction be on > the > > "RECEIVE" event as a metadata item specifying active vs passive. The > > protocol utilized as mentioned in the transport URI should clarify this > > though. > > > > In short - i think there is a way here that is all opt-in for existing > > users and components. > > > > Thanks > > > > On Tue, Oct 29, 2019 at 9:41 AM Nissim Shiman > > > wrote: > > > > > Adam, > > > good points... > > > I missed a step in explaining the use case where Provenance Events is > > > incomplete... > > > Where the second nifi does a GetSFTP from the *filesytem* that the > first > > > nifi is located on > > > So the second nifi currently sends a RECEIVE event, but there is no > &
Re: PULL ProvenanceEvent
Having an attribute added indicating passive/active/query for RECEIVE and FETCH will work, but nifi attributes are stateful (i.e. they will still be on the flowfile as metadata a couple of processor steps down the flow) Maybe an option is to expand the the api for RECEIVE and FETCH for with a new parameter for passive/active/query ? (i.e. the existing message signatures, such as [1] will remain the same, but new ones will be added to handle this new parameter? [1] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 On Thursday, October 31, 2019, 10:10:40 PM EDT, Joe Witt wrote: These distinctions may be meaningful. Adding them as an attribute lets the meaning convey but not introduce complexity for the majority case which is the distinction isnt key. thanks On Thu, Oct 31, 2019 at 4:05 PM Nissim Shiman wrote: > Mike, > I like the QUERY type as well. Basically a more refined PULL. Very nice. > > > Part of the challenge of adding PULL as a type is that there are currently > two flavors of RECEIVEs. > RECEIVE and FETCH [1] > > So any addition of a PULL would need a second flavor of PULL to match the > case where a flowfile's contents are being overwritten as well (i.e. as > FETCH is currently doing) > > > [1] > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java#L42 > > > Thanks, > Nissim > > > On Wednesday, October 30, 2019, 6:41:04 PM EDT, Mike Thomsen < > mikerthom...@gmail.com> wrote: > > I like the idea of creating PULL as a type. In fact, I'd propose that > there > are three scenarios here: > > RECEIVE - Passively acquire in a sort of hand-off situation. Ex: Kafka > subscription > PULL - Direct operations to seek out and fetch something in a targeted > fashion. Ex. GetHttp > QUERY - Go looking for the data and take what matches your search. Ex. > JsonQueryElasticsearch, GetMongo, any SQL query processor, etc. > > > > On Wed, Oct 30, 2019 at 1:31 PM Nissim Shiman > wrote: > > > Joe, > > > > > > It is hard to say how much value transit URI would bring to clarify a > > RECEIVE. > > For example a RECEIVE with transit URI of https: could be either a > > GetHTTP (i.e. active) or ListenHTTP (i.e. passive) > > > > but your idea of "a metadata item specifying active vs passive" is a very > > clever way to make this work with mimimal disruptions. > > > > My understanding of this is that the current receive() calls in > > ProvenanceReporter [1] will remain the same, but news ones will be added > > with a boolean parameter reflecting if the receive is active or passive. > > This will allow the current list of Provenance Events [2] to remain the > > same. So third party/custom processors can continue working as is > > > > Does this sound like what you are thinking? > > > > > > [1] > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > > [2] > > apache/nifi > > > > > > Thanks, > > > > Nissim > > On Tuesday, October 29, 2019, 12:47:40 PM EDT, Joe Witt < > > joe.w...@gmail.com> wrote: > > > > Nissim > > > > I like the idea to introduce a more refined type of event for how data is > > brought into nifi (active - PULL, passive - RECEIVE). > > > > That said it might be sufficient to simply have this distinction be on > the > > "RECEIVE" event as a metadata item specifying active vs passive. The > > protocol utilized as mentioned in the transport URI should clarify this > > though. > > > > In short - i think there is a way here that is all opt-in for existing > > users and components. > > > > Thanks > > > > On Tue, Oct 29, 2019 at 9:41 AM Nissim Shiman > > > wrote: > > > > > Adam, > > > good points... > > > I missed a step in explaining the use case where Provenance Events is > > > incomplete... > > > Where the second nifi does a GetSFTP from the *filesytem* that the > first > > > nifi is located on > > > So the second nifi currently sends a RECEIVE event, but there is no > > > corresponding SEND event from the first nifi (nor should there be) > > > If the second nifi sent a PULL event, it would be easier for a system > > > overseer to know that there should be no corresponding SEND event > > > > > > Currently send/receive works well when nifi 1 does a PostHTTP and nifi > 2 > > >
Re: PULL ProvenanceEvent
Mike, I like the QUERY type as well. Basically a more refined PULL. Very nice. Part of the challenge of adding PULL as a type is that there are currently two flavors of RECEIVEs. RECEIVE and FETCH [1] So any addition of a PULL would need a second flavor of PULL to match the case where a flowfile's contents are being overwritten as well (i.e. as FETCH is currently doing) [1] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java#L42 Thanks, Nissim On Wednesday, October 30, 2019, 6:41:04 PM EDT, Mike Thomsen wrote: I like the idea of creating PULL as a type. In fact, I'd propose that there are three scenarios here: RECEIVE - Passively acquire in a sort of hand-off situation. Ex: Kafka subscription PULL - Direct operations to seek out and fetch something in a targeted fashion. Ex. GetHttp QUERY - Go looking for the data and take what matches your search. Ex. JsonQueryElasticsearch, GetMongo, any SQL query processor, etc. On Wed, Oct 30, 2019 at 1:31 PM Nissim Shiman wrote: > Joe, > > > It is hard to say how much value transit URI would bring to clarify a > RECEIVE. > For example a RECEIVE with transit URI of https: could be either a > GetHTTP (i.e. active) or ListenHTTP (i.e. passive) > > but your idea of "a metadata item specifying active vs passive" is a very > clever way to make this work with mimimal disruptions. > > My understanding of this is that the current receive() calls in > ProvenanceReporter [1] will remain the same, but news ones will be added > with a boolean parameter reflecting if the receive is active or passive. > This will allow the current list of Provenance Events [2] to remain the > same. So third party/custom processors can continue working as is > > Does this sound like what you are thinking? > > > [1] > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > [2] > apache/nifi > > > Thanks, > > Nissim > On Tuesday, October 29, 2019, 12:47:40 PM EDT, Joe Witt < > joe.w...@gmail.com> wrote: > > Nissim > > I like the idea to introduce a more refined type of event for how data is > brought into nifi (active - PULL, passive - RECEIVE). > > That said it might be sufficient to simply have this distinction be on the > "RECEIVE" event as a metadata item specifying active vs passive. The > protocol utilized as mentioned in the transport URI should clarify this > though. > > In short - i think there is a way here that is all opt-in for existing > users and components. > > Thanks > > On Tue, Oct 29, 2019 at 9:41 AM Nissim Shiman > wrote: > > > Adam, > > good points... > > I missed a step in explaining the use case where Provenance Events is > > incomplete... > > Where the second nifi does a GetSFTP from the *filesytem* that the first > > nifi is located on > > So the second nifi currently sends a RECEIVE event, but there is no > > corresponding SEND event from the first nifi (nor should there be) > > If the second nifi sent a PULL event, it would be easier for a system > > overseer to know that there should be no corresponding SEND event > > > > Currently send/receive works well when nifi 1 does a PostHTTP and nifi 2 > > does a ListenHTTP, but not in the case above. > > > > The ERROR case you mention is a nice point as well, although not my > > specific issue at the moment. > > Thanks, > > Nissim > > > > > > > > > > > > On Monday, October 28, 2019, 11:52:57 PM EDT, Adam Taft < > > a...@adamtaft.com> wrote: > > > > > But a flowfile that was PULLed by the second nifi (from the first > nifi) > > will not necessarily have any provenance event generated by the first > nifi. > > > > Isn't this the fault of the first NiFi to fail to emit a SEND event in > > response to the second NiFi's request? In this scenario, shouldn't the > > send/receive pair be: > > NiFi-1 [SEND] :: NIFI-2 [RECEIVE]? > > > > What you describe is an odd use case for NiFi. NiFi is usually not in > the > > business of acting as a file server daemon in order to "send" flowfiles > to > > other systems. As you mention, HandleHttpResponse may be a lone wolf > > example processor which generates a SEND event whose input originates > from > > a "listener". [1] The other ListenXYZ processors generally issue RECEIVE > > events because they are receiving bytes, not generating them. > > > > Are there other processors in question? Something custom? Or is this > > re
Re: PULL ProvenanceEvent
Joe,
It is hard to say how much value transit URI would bring to clarify a RECEIVE.
For example a RECEIVE with transit URI of https: could be either a
GetHTTP (i.e. active) or ListenHTTP (i.e. passive)
but your idea of "a metadata item specifying active vs passive" is a very
clever way to make this work with mimimal disruptions.
My understanding of this is that the current receive() calls in
ProvenanceReporter [1] will remain the same, but news ones will be added with a
boolean parameter reflecting if the receive is active or passive.
This will allow the current list of Provenance Events [2] to remain the same.
So third party/custom processors can continue working as is
Does this sound like what you are thinking?
[1]
https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46
[2]
https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java
Thanks,
Nissim
On Tuesday, October 29, 2019, 12:47:40 PM EDT, Joe Witt
wrote:
Nissim
I like the idea to introduce a more refined type of event for how data is
brought into nifi (active - PULL, passive - RECEIVE).
That said it might be sufficient to simply have this distinction be on the
"RECEIVE" event as a metadata item specifying active vs passive. The
protocol utilized as mentioned in the transport URI should clarify this
though.
In short - i think there is a way here that is all opt-in for existing
users and components.
Thanks
On Tue, Oct 29, 2019 at 9:41 AM Nissim Shiman
wrote:
> Adam,
> good points...
> I missed a step in explaining the use case where Provenance Events is
> incomplete...
> Where the second nifi does a GetSFTP from the *filesytem* that the first
> nifi is located on
> So the second nifi currently sends a RECEIVE event, but there is no
> corresponding SEND event from the first nifi (nor should there be)
> If the second nifi sent a PULL event, it would be easier for a system
> overseer to know that there should be no corresponding SEND event
>
> Currently send/receive works well when nifi 1 does a PostHTTP and nifi 2
> does a ListenHTTP, but not in the case above.
>
> The ERROR case you mention is a nice point as well, although not my
> specific issue at the moment.
> Thanks,
> Nissim
>
>
>
>
>
> On Monday, October 28, 2019, 11:52:57 PM EDT, Adam Taft <
> a...@adamtaft.com> wrote:
>
> > But a flowfile that was PULLed by the second nifi (from the first nifi)
> will not necessarily have any provenance event generated by the first nifi.
>
> Isn't this the fault of the first NiFi to fail to emit a SEND event in
> response to the second NiFi's request? In this scenario, shouldn't the
> send/receive pair be:
> NiFi-1 [SEND] :: NIFI-2 [RECEIVE]?
>
> What you describe is an odd use case for NiFi. NiFi is usually not in the
> business of acting as a file server daemon in order to "send" flowfiles to
> other systems. As you mention, HandleHttpResponse may be a lone wolf
> example processor which generates a SEND event whose input originates from
> a "listener". [1] The other ListenXYZ processors generally issue RECEIVE
> events because they are receiving bytes, not generating them.
>
> Are there other processors in question? Something custom? Or is this
> related to site-to-site transfers?
>
> I still kind of question the motive of a provenance event pair that is
> trying to establish "who called who first". Honestly just trying to
> understand the use case where a matching SEND/RECEIVE pair doesn't give you
> what you need.
>
> The only thing I could see would be a processor that asks for data, but
> then doesn't receive it due to some error condition. In this case, adding
> some sort of ERROR event might be useful. "I attempted to retrieve data
> from ${uri}, but the transfer failed because of ${error condition}". That
> way, GetXYZ processors could report an error to provenance instead of as a
> bulletin.
>
> If the problem is related to a processor or the framework itself not
> generating an event, can we just fix that function to emit SEND in the
> scenario that you describe? Changing the provenance model itself (beyond
> possibly adding an ERROR event) feels like it would be the last scenario to
> consider.
>
> Thanks,
> Adam
>
> [1]
>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/HandleHttpResponse.java#L191
>
>
>
>
> On Mon, Oct 28, 2019 at 4:47 PM Nissim Shiman
> wrote:
>
> > Adam,
> > I believe there is a need for more detailed ProvenanceEve
Re: PULL ProvenanceEvent
Adam,
good points...
I missed a step in explaining the use case where Provenance Events is
incomplete...
Where the second nifi does a GetSFTP from the *filesytem* that the first nifi
is located on
So the second nifi currently sends a RECEIVE event, but there is no
corresponding SEND event from the first nifi (nor should there be)
If the second nifi sent a PULL event, it would be easier for a system overseer
to know that there should be no corresponding SEND event
Currently send/receive works well when nifi 1 does a PostHTTP and nifi 2 does a
ListenHTTP, but not in the case above.
The ERROR case you mention is a nice point as well, although not my specific
issue at the moment.
Thanks,
Nissim
On Monday, October 28, 2019, 11:52:57 PM EDT, Adam Taft
wrote:
> But a flowfile that was PULLed by the second nifi (from the first nifi)
will not necessarily have any provenance event generated by the first nifi.
Isn't this the fault of the first NiFi to fail to emit a SEND event in
response to the second NiFi's request? In this scenario, shouldn't the
send/receive pair be:
NiFi-1 [SEND] :: NIFI-2 [RECEIVE]?
What you describe is an odd use case for NiFi. NiFi is usually not in the
business of acting as a file server daemon in order to "send" flowfiles to
other systems. As you mention, HandleHttpResponse may be a lone wolf
example processor which generates a SEND event whose input originates from
a "listener". [1] The other ListenXYZ processors generally issue RECEIVE
events because they are receiving bytes, not generating them.
Are there other processors in question? Something custom? Or is this
related to site-to-site transfers?
I still kind of question the motive of a provenance event pair that is
trying to establish "who called who first". Honestly just trying to
understand the use case where a matching SEND/RECEIVE pair doesn't give you
what you need.
The only thing I could see would be a processor that asks for data, but
then doesn't receive it due to some error condition. In this case, adding
some sort of ERROR event might be useful. "I attempted to retrieve data
from ${uri}, but the transfer failed because of ${error condition}". That
way, GetXYZ processors could report an error to provenance instead of as a
bulletin.
If the problem is related to a processor or the framework itself not
generating an event, can we just fix that function to emit SEND in the
scenario that you describe? Changing the provenance model itself (beyond
possibly adding an ERROR event) feels like it would be the last scenario to
consider.
Thanks,
Adam
[1]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/HandleHttpResponse.java#L191
On Mon, Oct 28, 2019 at 4:47 PM Nissim Shiman
wrote:
> Adam,
> I believe there is a need for more detailed ProvenanceEvents.
> A use case would be a customer that is trying to track data passed between
> two nifi's and trying to match up SENDs and RECEIVEs
>
> So a flowfile that has a SEND event on the first nifi should have a
> RECEIVE event on the second nifi.
> But a flowfile that was PULLed by the second nifi (from the first nifi)
> will not necessarily have any provenance event generated by the first nifi.
>
> (I realize that FETCH is already a "reserved word" in the current
> ProvenanceEvents setup, so I was hoping PULL could be used instead.)
> There is another Provenance Event, ACKNOWLEDGE, which would also fit
> occasionally to this model as well (an example would be HandleHttpResponse
> processor which could send this instead of SEND when responding to a HTTP
> request)
> This being said, you make an excellent point when you said
> "However even more important to realize,
> this change would affect many other downstream consumers of provenance data
> which aren't necessarily in the stock NiFi distribution."
> Thanks,
> Nissim
>
> On Friday, October 11, 2019, 11:30:19 AM EDT, Nissim Shiman
> wrote:
>
> Adam,
> "Yes" to your first question and the four processor examples you listed.
>
> I will need to get back to you regarding your other points.
>
> Thanks,
> Nissim
>
> On Thursday, October 10, 2019, 7:05:57 PM EDT, Adam Taft <
> a...@adamtaft.com> wrote:
>
> Nissim,
>
> Just to be clear, you are trying to distinguish between processors which
> are actively "pulling" data (GetXYZ) vs. processors which just "listen" for
> data (ListenXYZ)? Is that your basic vision?
>
> GetFile => PULL
> GetHTTP => PULL
> ListenHTTP => RECEIVE
> ListenTCP => RECEIVE
>
> Could you clarify what advantages this would have in terms of data
> provenance? What would you use
Re: PULL ProvenanceEvent
Adam, I believe there is a need for more detailed ProvenanceEvents. A use case would be a customer that is trying to track data passed between two nifi's and trying to match up SENDs and RECEIVEs So a flowfile that has a SEND event on the first nifi should have a RECEIVE event on the second nifi. But a flowfile that was PULLed by the second nifi (from the first nifi) will not necessarily have any provenance event generated by the first nifi. (I realize that FETCH is already a "reserved word" in the current ProvenanceEvents setup, so I was hoping PULL could be used instead.) There is another Provenance Event, ACKNOWLEDGE, which would also fit occasionally to this model as well (an example would be HandleHttpResponse processor which could send this instead of SEND when responding to a HTTP request) This being said, you make an excellent point when you said "However even more important to realize, this change would affect many other downstream consumers of provenance data which aren't necessarily in the stock NiFi distribution." Thanks, Nissim On Friday, October 11, 2019, 11:30:19 AM EDT, Nissim Shiman wrote: Adam, "Yes" to your first question and the four processor examples you listed. I will need to get back to you regarding your other points. Thanks, Nissim On Thursday, October 10, 2019, 7:05:57 PM EDT, Adam Taft wrote: Nissim, Just to be clear, you are trying to distinguish between processors which are actively "pulling" data (GetXYZ) vs. processors which just "listen" for data (ListenXYZ)? Is that your basic vision? GetFile => PULL GetHTTP => PULL ListenHTTP => RECEIVE ListenTCP => RECEIVE Could you clarify what advantages this would have in terms of data provenance? What would you use this new event type for specifically? What are you missing now? Do you have a use case that needs this, or are you just generally trying to round out the provenance event types for sake of completeness? I honestly don't know a use case where you care whether you polled for the data or listened for it. The provenance model today just cares that you received the data, not so much how you received it. You're right that this proposal will affect many processors and the internal visualization tools, etc. However even more important to realize, this change would affect many other downstream consumers of provenance data which aren't necessarily in the stock NiFi distribution. For example, any third-party/custom ReportingTask that handles provenance data would need to be updated with this change. There's probably need for a strong vision to help demonstrate the value for this vs. the cost of the cascading effects related to this change. Thanks, Adam On Thu, Oct 10, 2019 at 4:02 PM Nissim Shiman wrote: > Hello Team, > > The ProvenanceEventType class does a good job capturing possible events, > but the PULL event doesn't seem to fall nicely into any of the existing > types. > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=blob;f=nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java > RECEIVE is the closest, but RECEIVE is passive and doesn't capture the > active action of a PULL > > And... maybe it would fall into FETCH, but FETCH is more focused on > contents of an existing flow file being overwritten. > > What does the community think about a new PULL event type, > or > using FETCH for PULL, and having what FETCH does now be a new event such > as REUSE > > NOTE: a new PULL event would have a cascading effect of many processors > that currently are emitting RECEIVE's being modified to be PULL > (i.e. So GetFile would no longer be a RECEIVE, but rather a PULL), but > would more accurately capture the event. > > Thanks, > Nissim Shiman > >
Re: PULL ProvenanceEvent
Adam, "Yes" to your first question and the four processor examples you listed. I will need to get back to you regarding your other points. Thanks, Nissim On Thursday, October 10, 2019, 7:05:57 PM EDT, Adam Taft wrote: Nissim, Just to be clear, you are trying to distinguish between processors which are actively "pulling" data (GetXYZ) vs. processors which just "listen" for data (ListenXYZ)? Is that your basic vision? GetFile => PULL GetHTTP => PULL ListenHTTP => RECEIVE ListenTCP => RECEIVE Could you clarify what advantages this would have in terms of data provenance? What would you use this new event type for specifically? What are you missing now? Do you have a use case that needs this, or are you just generally trying to round out the provenance event types for sake of completeness? I honestly don't know a use case where you care whether you polled for the data or listened for it. The provenance model today just cares that you received the data, not so much how you received it. You're right that this proposal will affect many processors and the internal visualization tools, etc. However even more important to realize, this change would affect many other downstream consumers of provenance data which aren't necessarily in the stock NiFi distribution. For example, any third-party/custom ReportingTask that handles provenance data would need to be updated with this change. There's probably need for a strong vision to help demonstrate the value for this vs. the cost of the cascading effects related to this change. Thanks, Adam On Thu, Oct 10, 2019 at 4:02 PM Nissim Shiman wrote: > Hello Team, > > The ProvenanceEventType class does a good job capturing possible events, > but the PULL event doesn't seem to fall nicely into any of the existing > types. > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=blob;f=nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java > RECEIVE is the closest, but RECEIVE is passive and doesn't capture the > active action of a PULL > > And... maybe it would fall into FETCH, but FETCH is more focused on > contents of an existing flow file being overwritten. > > What does the community think about a new PULL event type, > or > using FETCH for PULL, and having what FETCH does now be a new event such > as REUSE > > NOTE: a new PULL event would have a cascading effect of many processors > that currently are emitting RECEIVE's being modified to be PULL > (i.e. So GetFile would no longer be a RECEIVE, but rather a PULL), but > would more accurately capture the event. > > Thanks, > Nissim Shiman > >
PULL ProvenanceEvent
Hello Team, The ProvenanceEventType class does a good job capturing possible events, but the PULL event doesn't seem to fall nicely into any of the existing types. https://gitbox.apache.org/repos/asf?p=nifi.git;a=blob;f=nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java RECEIVE is the closest, but RECEIVE is passive and doesn't capture the active action of a PULL And... maybe it would fall into FETCH, but FETCH is more focused on contents of an existing flow file being overwritten. What does the community think about a new PULL event type, or using FETCH for PULL, and having what FETCH does now be a new event such as REUSE NOTE: a new PULL event would have a cascading effect of many processors that currently are emitting RECEIVE's being modified to be PULL (i.e. So GetFile would no longer be a RECEIVE, but rather a PULL), but would more accurately capture the event. Thanks, Nissim Shiman
