Re: Jira contributor access
Hi Cory, Thanks for submitting the issue and requesting access to Jira, I'm sure one of the project maintainers will be able to add you soon. I added a comment to the issue you submitted and I would be interested to hear more details on your proposed implementation. As mentioned in the comment, I have another open issue that is closely related to S/MIME handling, and includes some Controller Services that could be useful in implementing signed S/MIME emails. Feel free to provide your comments on the related issue and PR in GitHub. Regards, David Handermann On Tue, Dec 1, 2020 at 10:00 PM cWix Software wrote: > Hi there, > > I would like to work on Jira ticket NIFI-8059. > > I was reading you had to give me permissions so I could. My login is > cwixom. > > Please let me know if there's anything else I need to provide. > > Thanks! > > Cory Wixom >
Re: [VOTE] Release Apache NiFi 1.13.0
-1 non-binding. Verified release with successful build on Azul Zulu JDK 11.0.10 on Ubuntu 20.0.10. Verified sample flow with InvokeHTTP and ListenHTTP processors using multiple keystore types and TLS configuration options. Unfortunately found bcprov-ext-jdk15on-1.60.jar together with bcprov-jdk15on-1.68.jar in nifi-framework-nar. The bcprov-ext-jdk15on-1.60.jar library is apparently a transitive dependency of spring-security-saml2-core through a library named com.narupley:not-going-to-be-commons-ssl. The Bouncy Castle libraries should be version 1.68 throughout the NiFi framework. The bcprov-ext-jdk15on library contains the same classes as bcprov-jdk15on plus a handful of additional classes for infrequently used algorithms. The presence of both versions did not appear to cause problems during initial tests, but it could cause unexpected behavior at runtime depending on which version gets loaded. If the Spring Security SAML2 library requires the algorithms present in bcprov-ext-jdk15on, it will probably be necessary to change dependencies in NiFi to replace references to bcprov-jdk15on with bcprov-ext-jdk15on to ensure a consistent version and avoid duplication. Regards, David Handermann On Fri, Jan 29, 2021 at 5:33 PM M Tien wrote: > +1 non-binding. > > Went through the release guide > Verified a full build on JDK 1.8.0_275 and JDK 11.0.5 > Verified a secure instance of NiFi > Verified I was able to authenticate with OIDC using Google, Okta, and > Azure and I can successfully log out and invalidate the JWT. > > - Margot
Re: [VOTE] Release Apache NiFi 1.13.0 (rc2)
+1 non-binding Verified release signatures and expected files. Verified build on Ubuntu 20.10 using Apache Maven 3.6.3 with Azul Zulu JDK 11.0.10 and AdoptOpenJDK 1.8.0_282. Tested sample flows with ListenHTTP and InvokeHTTP using different TLS versions and keystore types to verify BCFKS support as well as TLS protocol support differences on Java 8 and 11. Confirmed absence of bcprov-ext-jdk15on from nifi-framework-nar as documented in NIFI-8186. Regards, David Handermann On Tue, Feb 2, 2021 at 12:18 PM Mark Payne wrote: > +1 (binding) > > Build details: > Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec > Java version: 11.0.8, vendor: AdoptOpenJDK, runtime: > /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home > Default locale: en_US, platform encoding: UTF-8 > OS name: "mac os x", version: "10.15.6", arch: "x86_64", family: “mac" > > Verified hashes, README, NOTICE, and LICENSE files look as expected. > > Performed some smoke tests. Verified new dependent properties feature. > Exported flow and ran it with bin/nifi.sh stateless. > > Thanks for putting together the release, Joe! > > > > > On Feb 2, 2021, at 12:32 PM, Peter Turcsanyi > wrote: > > > > +1 non-binding > > > > Went through the release helper guide. > > Verified full build with empty local maven repo with Java 8 (AdoptOpenJDK > > 1.8.0_282-b08) and 11 (AdoptOpenJDK 11.0.10+9) on Ubuntu 20.04. > > Ran full builds with different time zone settings to verify NIFI-8023, > also > > tested it with flows. > > Ran some SSL related test flows with processors changed recently > > (ListenHTTP, GRPC, MQTT). > > > > Thanks for RMing Joe! > > > > On Tue, Feb 2, 2021 at 6:04 PM Otto Fowler > wrote: > > > >> +1 > >> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > >> Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec > >> Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: > >> /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > >> Default locale: en_US, platform encoding: UTF-8 > >> OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac" > >> > >> Thanks Joe! > >> > >> > >>> On Feb 1, 2021, at 20:10, Joe Witt wrote: > >>> > >>> Hello, > >>> > >>> I am pleased to be calling this vote for the source release of Apache > >> NiFi > >>> 1.13.0. > >>> > >>> The source zip, including signatures, digests, etc. can be found at: > >>> https://repository.apache.org/content/repositories/orgapachenifi-1176 > >>> > >>> The source being voted upon and the convenience binaries can be found > at: > >>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/ > >>> > >>> A helpful reminder on how the release candidate verification process > >> works: > >>> > >> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > >>> > >>> The Git tag is nifi-1.13.0-RC2 > >>> The Git commit ID is c27e59fc679a2e982102a75b8b8df2b0f062af23 > >>> > >> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=c27e59fc679a2e982102a75b8b8df2b0f062af23 > >>> > >>> Checksums of nifi-1.13.0-source-release.zip: > >>> SHA256: > 4913dd3d943afac710d1a277bf31beebf7a6207a20e1148849d69511f44fc97b > >>> SHA512: > >>> > >> > dc9935f0eb8692cd8493f5863bc8ae2ef0b52653fa69ff8b9a7e8db7dbd9ec6561f6ffdca4a1b55e43b289d04f5671f5ab4de30999838c5fca5c282c00a7c7f8 > >>> > >>> Release artifacts are signed with the following key: > >>> https://people.apache.org/keys/committer/joewitt.asc > >>> > >>> KEYS file available here: > >>> https://dist.apache.org/repos/dist/release/nifi/KEYS > >>> > >>> 238 issues were closed/resolved for this release: > >>> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12348700 > >>> > >>> Release note highlights can be found here: > >>> > >> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0 > >>> > >>> The vote will be open for 72 hours. > >>> Please download the release candidate and evaluate the necessary items > >>> including checking hashes, signatures, build from source, and test. > Then > >>> please vote: > >>> > >>> [ ] +1 Release this package as nifi-1.13.0 > >>> [ ] +0 no opinion > >>> [ ] -1 Do not release this package because... > >> > >> > >
Re: [discuss] we need to enable secure by default...
Having NiFi enforce authentication by default seems like the right way to go, especially given the capabilities of the system. Bryan raises a good point about storage of account information, so weighing the positives and negatives of various identity providers should be considered. Following on Joe's point about disabling plain HTTP, one option could be generating both client and server certificates and using Mutual TLS for authentication. This would obviously require installing the client certificate in a browser, which is not trivial, but could be part of an installation guide. This approach definitely provides a high barrier of entry of new users, but provides a strong level of security while avoiding the need for some other identity provider service to be configured. As others have mentioned, this seems necessary to address the situation of someone installing NiFi without a full understanding of the configuration required, so it is important to keep the audience in mind when evaluating a solution. Regards, David Handermann On Wed, Feb 10, 2021 at 7:39 AM Bryan Bende wrote: > Just to clarify, I was not suggesting that we make a default secure > setup that requires LDAP. > > I was just saying that in order to provide a default secure setup, > we'd have to provide a login identity provider implementation that is > backed by a file or database table, which in the past we decided > against. > > On Wed, Feb 10, 2021 at 8:28 AM Russell Bateman > wrote: > > > > I second the concerns expressed, but second especially Bryan's pointing > > out that requiring LDAP/AD to be set up in order even to begin to use > > our framework would be a bit onerous for developers just interested in > > getting work done and a barrier to considering the framework should it > > be erected a little too high. Should we at least glance at how this is > > solved by the likes of other projects, Kafka and Cassandra come to mind, > > even if it means resorting to a store of a name or two? I didn't find > > getting into developing with them a pain, but making me jump through the > > hoop of setting up LDAP may very well have changed that. > > > > These unsecure instances of NiFi out there are not our community's > > fault. I suppose we're worried about getting splattered by bad press? > > > > On 2/10/21 5:47 AM, Bryan Bende wrote: > > > I agree with the overall idea, although I would think it requires a > > > major release to make this kind of change to the default behavior. > > > > > > Also, we have always avoided NiFi being a store of usernames and > > > passwords, so we don't have a login provider that uses a local file or > > > a database, we've always said you connect to LDAP/AD for that. > > > > > > Obviously it can be implemented, but just pointing out that we'd have > > > to change our stance here if we want to provide a default username and > > > password to authenticate with. > > > > > > On Tue, Feb 9, 2021 at 11:25 PM Andrew Grande > wrote: > > >> Mysql has been generating an admin password on default installs for, > like, > > >> forever. This workflow should be familiar for many users. > > >> > > >> I'd suggest taking the automation tooling into account and how a > production > > >> rollout (user-provided password) would fit into the workflow. > > >> > > >> Andrew > > >> > > >> On Tue, Feb 9, 2021, 8:15 PM Tony Kurc wrote: > > >> > > >>> Joe, > > >>> In addition to your suggestions, were you thinking of making this > processor > > >>> disabled by default as well? > > >>> > > >>> Tony > > >>> > > >>> > > >>> On Tue, Feb 9, 2021, 11:04 PM Joe Witt wrote: > > >>> > > >>>> Team > > >>>> > > >>>> While secure by default may not be practical perhaps ‘not blatantly > wide > > >>>> open’ by default should be adopted. > > >>>> > > >>>> I think we should consider killing support for http entirely and > support > > >>>> only https. We should consider auto generating a user and password > and > > >>>> possibly server cert if nothing is configured and log the generated > user > > >>>> and password. Sure it could still be configured to be non secure > but > > >>> that > > >>>> would truly be an admins fault. Now its just ‘on’ > > >>>> > > >>>> This tweet is a great example of why > > >>>> > > >>>> https://twitter.com/_escctrl_/status/1359280656174510081?s=21 > > >>>> > > >>>> > > >>>> Who agrees? Who disagrees? Please share ideas. > > >>>> > > >>>> Thanks > > >>>> >
Re: [discuss] we need to enable secure by default...
Mark, Thanks for your perspective on certificate generation, I agree that troubleshooting certificates can be confusing due to unclear error messaging. Generating self-signed certificates for one-way TLS still requires the user to accept the certificate presented, but at least that is more common in various products. Are you concerned about that situation, or are you more concerned about the additional challenges of mutual TLS authentication? Generating a random password in absence of any other configuration would certainly be easier from a new user perspective. Some of the security concerns with password authentication can be mitigated with one-way TLS, so a blending of these approaches, as Joe describes in NIFI-8220, seems like a good way to go. Regards, David Handermann On Wed, Feb 10, 2021 at 8:23 AM Mark Payne wrote: > I would be in favor of this as well. I agree that there is no need to wait > for a 2.0 version - there is no inherit backward compatibility challenge > here. > Requiring that new application configs be set, etc. I think is completely > fair game between minor versions. > > Personally, though, I would very much stray away from auto-generating > certificates. For those of us who have dealt with certificates significantly > In the past, minor configuration issues are easy to address. But for > someone who hasn’t spent much time dealing with certificates, the error > messages > that are often intentionally vague can quickly result in users being > overwhelmed. This is especially true if browser configurations are already > setup to > automatically offer certificates that area already installed - this can > result in weird error messages about untrusted certificates when the user > thinks > they haven’t even provided a certificate, etc. It gets really hairy. > > I am more in favor of a default username/password personally. It would > require implementing a new auth provider. And it’s one that historically we > have > avoided implementing because a basic auth type of approach is less secure > than mutual auth, ldap, etc. But it’s more secure than nothing. > > Thanks > -Mark > > > > On Feb 10, 2021, at 9:13 AM, Joe Witt wrote: > > > > There are a range of things to consider. And yes whatever we do will > > involve writing code. We also have to find the right place for the bar > > here. > > > > 1. Disable HTTP by default. And if they want to enable HTTP they should > > also have to make a config change indicating they are fully willing to > > accept that it is an entirely non secure configuration as far as the > > application is concerned. > > 2. Provide a default username/password provider out of the box configured > > by default and an auto generate self-signed server cert. This means the > > NiFi server will provide the client browser a cert. It won't be > > known/trusted so the browser will advise of this. And on startup NiFi > can > > auto generate and log this username and password. It would truly be a > > single username/password and not some store for various users. We > > disallow access to all restricted components by default. > > > > This default configuration at least means someone cannot start NiFi and > it > > is totally exposed by default while also preserving a pretty simple 'get > > started' model for the user. They'd have to take action to make it less > > secure. > > > > The option DavidH mentions of mutual auth could work well also and as he > > mentioned avoids the need for anything special in terms of auth provider > > which is compelling for us but I do worry about the user experience on > that. > > > > I'll add to this that I think we cannot afford to wait for a NiFi 2.0 > > release to take action here. Or that we should expedite a NiFi 2.0 > release > > to take action and that we should just not make the bar for what 2.0 is > so > > high that we never get this done. But frankly I think we could make this > > change in NiFi 1.15 and document what is happening. For existing > > installs/configs we'd not be changing anything except maybe when they're > > using HTTP and don't set the 'no seriously i want this thing wide open > > option'. > > > > Thanks > > > > On Wed, Feb 10, 2021 at 6:58 AM David Handermann < > exceptionfact...@gmail.com> > > wrote: > > > >> Having NiFi enforce authentication by default seems like the right way > to > >> go, especially given the capabilities of the system. > >> > >> Bryan raises a good point about storage of account information, so > weighing > >> the positives
Re: [discuss] we need to enable secure by default...
Mark, Thanks for clarifying, that makes sense. Regards, David Handermann On Wed, Feb 10, 2021 at 8:41 AM Mark Payne wrote: > David, > > My concern was purely around generating client certs and using mutual TLS. > I definitely think we should have a server cert if using username & > password. > > Thanks > -Mark > > > > On Feb 10, 2021, at 9:37 AM, David Handermann < > exceptionfact...@gmail.com> wrote: > > > > Mark, > > > > Thanks for your perspective on certificate generation, I agree that > > troubleshooting certificates can be confusing due to unclear error > > messaging. Generating self-signed certificates for one-way TLS still > > requires the user to accept the certificate presented, but at least that > is > > more common in various products. Are you concerned about that situation, > > or are you more concerned about the additional challenges of mutual TLS > > authentication? > > > > Generating a random password in absence of any other configuration would > > certainly be easier from a new user perspective. Some of the security > > concerns with password authentication can be mitigated with one-way TLS, > so > > a blending of these approaches, as Joe describes in NIFI-8220, seems > like a > > good way to go. > > > > Regards, > > David Handermann > > > > On Wed, Feb 10, 2021 at 8:23 AM Mark Payne wrote: > > > >> I would be in favor of this as well. I agree that there is no need to > wait > >> for a 2.0 version - there is no inherit backward compatibility challenge > >> here. > >> Requiring that new application configs be set, etc. I think is > completely > >> fair game between minor versions. > >> > >> Personally, though, I would very much stray away from auto-generating > >> certificates. For those of us who have dealt with certificates > significantly > >> In the past, minor configuration issues are easy to address. But for > >> someone who hasn’t spent much time dealing with certificates, the error > >> messages > >> that are often intentionally vague can quickly result in users being > >> overwhelmed. This is especially true if browser configurations are > already > >> setup to > >> automatically offer certificates that area already installed - this can > >> result in weird error messages about untrusted certificates when the > user > >> thinks > >> they haven’t even provided a certificate, etc. It gets really hairy. > >> > >> I am more in favor of a default username/password personally. It would > >> require implementing a new auth provider. And it’s one that > historically we > >> have > >> avoided implementing because a basic auth type of approach is less > secure > >> than mutual auth, ldap, etc. But it’s more secure than nothing. > >> > >> Thanks > >> -Mark > >> > >> > >>> On Feb 10, 2021, at 9:13 AM, Joe Witt wrote: > >>> > >>> There are a range of things to consider. And yes whatever we do will > >>> involve writing code. We also have to find the right place for the bar > >>> here. > >>> > >>> 1. Disable HTTP by default. And if they want to enable HTTP they > should > >>> also have to make a config change indicating they are fully willing to > >>> accept that it is an entirely non secure configuration as far as the > >>> application is concerned. > >>> 2. Provide a default username/password provider out of the box > configured > >>> by default and an auto generate self-signed server cert. This means > the > >>> NiFi server will provide the client browser a cert. It won't be > >>> known/trusted so the browser will advise of this. And on startup NiFi > >> can > >>> auto generate and log this username and password. It would truly be a > >>> single username/password and not some store for various users. We > >>> disallow access to all restricted components by default. > >>> > >>> This default configuration at least means someone cannot start NiFi and > >> it > >>> is totally exposed by default while also preserving a pretty simple > 'get > >>> started' model for the user. They'd have to take action to make it > less > >>> secure. > >>> > >>> The option DavidH mentions of mutual auth could work well also and as > he > >>> mentioned avoids the need for anything spec
Re: [discuss] we need to enable secure by default...
I agree that a generated password is the way to go, and the challenge is making it available to the user. Depending on how it is stored for the identity provider, having a command line tool to read and print it could be a reasonable option. Although this particular thread referenced a specific Twitter post, this general discussion is more of a reflection of the need to make things more secure by default as other products have followed similar approaches. Regards, David Handermann On Wed, Feb 10, 2021 at 8:53 AM Kevin Doran wrote: > I am in favor of requiring some authentication by default. > > I’m in favor of an admin username and generated password. (It sounds li9ke > most of us are on the same page, but I don’t think a static, default > password buys us much against the types of abuse shown on the twitter > thread Joe shared.) > > We need some way of making the generated password discoverable… Print to > the logs on first boot? Not great but are there other mechanisms? A setup > CLI utility? > > To help not impede automated deployments, maybe we should change the > startup flow such that there is a way to provide this password. That would > be better than people just disabling whatever default authentication we set. > > > > On Feb 10, 2021, at 09:45, David Handermann > wrote: > > > > Mark, > > > > Thanks for clarifying, that makes sense. > > > > Regards, > > David Handermann > > > > On Wed, Feb 10, 2021 at 8:41 AM Mark Payne wrote: > > > >> David, > >> > >> My concern was purely around generating client certs and using mutual > TLS. > >> I definitely think we should have a server cert if using username & > >> password. > >> > >> Thanks > >> -Mark > >> > >> > >>> On Feb 10, 2021, at 9:37 AM, David Handermann < > >> exceptionfact...@gmail.com> wrote: > >>> > >>> Mark, > >>> > >>> Thanks for your perspective on certificate generation, I agree that > >>> troubleshooting certificates can be confusing due to unclear error > >>> messaging. Generating self-signed certificates for one-way TLS still > >>> requires the user to accept the certificate presented, but at least > that > >> is > >>> more common in various products. Are you concerned about that > situation, > >>> or are you more concerned about the additional challenges of mutual TLS > >>> authentication? > >>> > >>> Generating a random password in absence of any other configuration > would > >>> certainly be easier from a new user perspective. Some of the security > >>> concerns with password authentication can be mitigated with one-way > TLS, > >> so > >>> a blending of these approaches, as Joe describes in NIFI-8220, seems > >> like a > >>> good way to go. > >>> > >>> Regards, > >>> David Handermann > >>> > >>> On Wed, Feb 10, 2021 at 8:23 AM Mark Payne > wrote: > >>> > >>>> I would be in favor of this as well. I agree that there is no need to > >> wait > >>>> for a 2.0 version - there is no inherit backward compatibility > challenge > >>>> here. > >>>> Requiring that new application configs be set, etc. I think is > >> completely > >>>> fair game between minor versions. > >>>> > >>>> Personally, though, I would very much stray away from auto-generating > >>>> certificates. For those of us who have dealt with certificates > >> significantly > >>>> In the past, minor configuration issues are easy to address. But for > >>>> someone who hasn’t spent much time dealing with certificates, the > error > >>>> messages > >>>> that are often intentionally vague can quickly result in users being > >>>> overwhelmed. This is especially true if browser configurations are > >> already > >>>> setup to > >>>> automatically offer certificates that area already installed - this > can > >>>> result in weird error messages about untrusted certificates when the > >> user > >>>> thinks > >>>> they haven’t even provided a certificate, etc. It gets really hairy. > >>>> > >>>> I am more in favor of a default username/password personally. It would > >>>> require implementing a new auth provider. And it’s one that > >> historically we > >>>> h
Re: [discuss] we need to enable secure by default...
Integrating an option to use Let's Encrypt would be a great improvement over self-signed certificates, but it is difficult to support that for instances of NiFi running on servers without Internet access. Even when running NiFi for local development purposes, the development system is not likely to have a publicly addressable DNS name, so Let's Encrypt is not an option in those cases. Regards, David Handermann On Wed, Feb 10, 2021 at 11:09 AM Joe Witt wrote: > Otto > > Installers like you mention are inherently brutal for portability so very > difficult for us in the community. From the vendor world we of course do > things like that. I think in apache nifi land we need a default 'even for > devs' which is not wide open. > > James > > I do think adding such a warning is good. But it doesn't help avoid these > wildly abusive cases. We need a secure by default path. We can probably > do better than the self signed path if we add a 'before running' step in > the CLI for the user. Not sure. > > Thanks > > On Wed, Feb 10, 2021 at 10:05 AM James Srinivasan < > james.sriniva...@gmail.com> wrote: > > > Would a suitably large warning on the http ui be a good starting point? > > > > Browsers are getting increasingly wary of self signed certs and we > probably > > don't want to be encouraging people to ignore them. > > > > What about easier acme+certbot support? (notwithstanding all the non > public > > deployments) > > > > On Wed, 10 Feb 2021, 15:25 Otto Fowler, wrote: > > > > > Aren’t most of these applications installed by an installer? > > > Maybe we can have one or more installers that setup a secure instance, > > and > > > those installers > > > could be the *production* nifi, and keep the zip distribution open for > > > developers? > > > > > > > > > > On Feb 10, 2021, at 10:04, David Handermann < > > exceptionfact...@gmail.com> > > > wrote: > > > > > > > > I agree that a generated password is the way to go, and the challenge > > is > > > > making it available to the user. Depending on how it is stored for > the > > > > identity provider, having a command line tool to read and print it > > could > > > be > > > > a reasonable option. > > > > > > > > Although this particular thread referenced a specific Twitter post, > > this > > > > general discussion is more of a reflection of the need to make things > > > more > > > > secure by default as other products have followed similar approaches. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Wed, Feb 10, 2021 at 8:53 AM Kevin Doran > wrote: > > > > > > > >> I am in favor of requiring some authentication by default. > > > >> > > > >> I’m in favor of an admin username and generated password. (It sounds > > > li9ke > > > >> most of us are on the same page, but I don’t think a static, default > > > >> password buys us much against the types of abuse shown on the > twitter > > > >> thread Joe shared.) > > > >> > > > >> We need some way of making the generated password discoverable… > Print > > to > > > >> the logs on first boot? Not great but are there other mechanisms? A > > > setup > > > >> CLI utility? > > > >> > > > >> To help not impede automated deployments, maybe we should change the > > > >> startup flow such that there is a way to provide this password. That > > > would > > > >> be better than people just disabling whatever default authentication > > we > > > set. > > > >> > > > >> > > > >>> On Feb 10, 2021, at 09:45, David Handermann < > > > exceptionfact...@gmail.com> > > > >> wrote: > > > >>> > > > >>> Mark, > > > >>> > > > >>> Thanks for clarifying, that makes sense. > > > >>> > > > >>> Regards, > > > >>> David Handermann > > > >>> > > > >>> On Wed, Feb 10, 2021 at 8:41 AM Mark Payne > > > wrote: > > > >>> > > > >>>> David, > > > >>>> > > > >>>> My concern was purely around generating client certs and using > > mutual > > > >> TLS. &
Re: [VOTE] Release Apache NiFi 1.13.0 (rc4)
+1 non-binding Verified release signatures and expected files. Verified build on Ubuntu 20.10 using Apache Maven 3.6.3 with Azul Zulu JDK 11.0.10 and AdoptOpenJDK 1.8.0_282. Verified absence of startup script shell warnings on Ubuntu 20.10. Verified service listening on localhost when using default nifi.properties. Configured and tested mutual TLS authentication on a standalone server with BCFKS key store and trust store. Regards, David Handermann On Fri, Feb 12, 2021 at 5:45 PM Muazma Zahid wrote: > +1 (non-binding) > > - Ran build with OpenJDK 1.8.0_275 on Linux > - Deployed cluster on Azure and tested flows with Blob, ADLS, and CosmosDB > processors. > > Looks good to me. > > Thanks > Muazma > > On Fri, Feb 12, 2021 at 3:38 PM Sushil Kumar wrote: > > > +1 (non-binding) Release this package as nifi-1.13.0 > > > > Deployed this via helm chart(https://github.com/sushilkm/nifi-chart) on > > kubernetes. > > Thank you to all the contributors and reviewers. > > > > On Fri, Feb 12, 2021 at 2:13 PM Marc Parisi wrote: > > > > > +1 > > > - Verified sigs and hashes > > > - Built on PopOS w/ java 8 && 11 > > > - Tested with basic flow sending data to various devices w/ Secured > > > transport > > > - Tested secured w/ secured nifi reg. > > > > > > Thanks, > > > Marc > > > > > > On Fri, Feb 12, 2021 at 2:56 PM Andrew Lim > > > > wrote: > > > > > > > +1 (binding) > > > > > > > > -Tested secure NiFi with secure NiFi Registry 0.8.0 > > > > -Ran basic flows successfully > > > > -Tested basic versioned flow functionality > > > > -Reviewed and tested Core UI fixes and Documentation updates > > > > > > > > Drew > > > > > > > > > On Feb 10, 2021, at 11:37 PM, Joe Witt wrote: > > > > > > > > > > Hello, > > > > > > > > > > I am pleased to be calling this vote for the source release of > Apache > > > > NiFi > > > > > 1.13.0. > > > > > > > > > > The source zip, including signatures, digests, etc. can be found > at: > > > > > > > https://repository.apache.org/content/repositories/orgapachenifi-1178 > > > > > > > > > > The source being voted upon and the convenience binaries can be > found > > > at: > > > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/ > > > > > > > > > > A helpful reminder on how the release candidate verification > process > > > > works: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > > > > > > > The Git tag is nifi-1.13.0-RC4 > > > > > The Git commit ID is 3bc6a122091214b33eee17a270163d7ca26e2a0c > > > > > > > > > > > > > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=3bc6a122091214b33eee17a270163d7ca26e2a0c > > > > > > > > > > Checksums of nifi-1.13.0-source-release.zip: > > > > > SHA256: > > > 95efe5db38e973c9f4062a7b2c95fdc5dad31d7c5e1d36ce1776b9b227c89b9c > > > > > SHA512: > > > > > > > > > > > > > > > d7dd9e851341ebd605784142a7861935f6f814bc612499013456a15713bc9119e426df8f26445c260bdb25cbfc21822cf0d44985bf372a065c9d8597953a3c4a > > > > > > > > > > Release artifacts are signed with the following key: > > > > > https://people.apache.org/keys/committer/joewitt.asc > > > > > > > > > > KEYS file available here: > > > > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > > > > > > > 260 issues were closed/resolved for this release: > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12348700 > > > > > > > > > > Release note highlights can be found here: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0 > > > > > > > > > > The vote will be open for 72 hours. > > > > > Please download the release candidate and evaluate the necessary > > items > > > > > including checking hashes, signatures, build from source, and test. > > > Then > > > > > please vote: > > > > > > > > > > [ ] +1 Release this package as nifi-1.13.0 > > > > > [ ] +0 no opinion > > > > > [ ] -1 Do not release this package because... > > > > > > > > > > > > > >
Re: [VOTE] Release Apache NiFi 1.13.1
+1 non-binding Verified release signatures and expected files. Verified build on Ubuntu 20.10 using Apache Maven 3.6.3 with Azul Zulu JDK 11.0.10. Configured and tested InvokeHTTP with multiple configurations including disabling HTTP/2. Regards, David Handermann On Fri, Mar 12, 2021 at 2:19 PM Joey Frazee wrote: > +1 (non-binding) > > - Verified checksums and signatures > - Ran full build on Java 1.8 and 11 on Linux > - Verified NIFI-3383, NIFI-8231, and NIFI-8200 > > -joey > > > On Mar 12, 2021, at 10:28 AM, Bryan Bende wrote: > > > > +1 (binding) > > > > - Verified everything in the standard release helper > > - Setup secure standalone instance with SAML authentication > > > >> On Fri, Mar 12, 2021 at 10:17 AM Marton Szasz > wrote: > >> > >> +1 (non-binding) > >> > >> Followed the release helper guide, tested the binary with a simple flow. > >> > >> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > >> Maven home: /usr/share/maven-bin-3.6 > >> Java version: 1.8.0_252, vendor: IcedTea, runtime: > /opt/icedtea-bin-3.16.0/jre > >> Default locale: en_US, platform encoding: UTF-8 > >> OS name: "linux", version: "5.10.12-gentoo-x86_64", arch: "amd64", > >> family: "unix" > >> > >> > >>> On Fri, 12 Mar 2021 at 13:35, Otto Fowler > wrote: > >>> > >>> +1 > >>> > >>> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > >>> Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec > >>> Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > >>> Default locale: en_US, platform encoding: UTF-8 > >>> OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac" > >>> > >>> > >>>> On Mar 11, 2021, at 11:29, Joe Witt wrote: > >>>> > >>>> Hello, > >>>> > >>>> I am pleased to be calling this vote for the source release of Apache > NiFi > >>>> 1.13.1. > >>>> > >>>> The source zip, including signatures, digests, etc. can be found at: > >>>> https://repository.apache.org/content/repositories/orgapachenifi-1179 > >>>> > >>>> The source being voted upon and the convenience binaries can be found > at: > >>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.1/ > >>>> > >>>> A helpful reminder on how the release candidate verification process > works: > >>>> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > >>>> > >>>> The Git tag is nifi-1.13.1-RC1 > >>>> The Git commit ID is acbc217cb7002d7489421f4d346b995a43b6ea01 > >>>> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=acbc217cb7002d7489421f4d346b995a43b6ea01 > >>>> > >>>> Checksums of nifi-1.13.1-source-release.zip: > >>>> SHA256: > 0a397df640e579720e26699e38a3738c5be05af01aad8aaeefc04eb58591faac > >>>> SHA512: > >>>> > 7f8df759d4345ccd6e75c169bd0aab1b7f4f64bf5a8b11b45bc1d7c8163acd0035922dcdbef232392279f4ea0710d4a97c55d480281bfe1d50b6401295633d48 > >>>> > >>>> Release artifacts are signed with the following key: > >>>> https://people.apache.org/keys/committer/joewitt.asc > >>>> > >>>> KEYS file available here: > >>>> https://dist.apache.org/repos/dist/release/nifi/KEYS > >>>> > >>>> 48 issues were closed/resolved for this release: > >>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12348700 > >>>> > >>>> Release note highlights can be found here: > >>>> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1 > >>>> > >>>> The vote will be open for 72 hours. > >>>> Please download the release candidate and evaluate the necessary items > >>>> including checking hashes, signatures, build from source, and test. > Then > >>>> please vote: > >>>> > >>>> [ ] +1 Release this package as nifi-1.13.1 > >>>> [ ] +0 no opinion > >>>> [ ] -1 Do not release this package because... > >>> >
Re: [VOTE] Release Apache NiFi 1.13.2
+1 non-binding - Verified release signatures and expected files - Verified build on Ubuntu 20.10 using Apache Maven 3.6.3 with Azul Zulu JDK 11.0.10 - Verified Admin Guide update for NIFI-8324 - Verified valid JSON output from SplitJson for NIFI-8342 Regards, David On Thu, Mar 18, 2021 at 1:09 PM Pierre Villard wrote: > +1 binding > > Confirmed the main fix with a flow provided in the JIRA. > > Thanks, > Pierre > > Le jeu. 18 mars 2021 à 21:37, Mark Payne a écrit : > > > +1 (binding) > > > > Was able to build, and successfully run all system tests, including those > > added for NIFI-8337. Also started up and manually verified that NIFI-8337 > > has been addressed. > > > > Thanks > > -Mark > > > > > On Mar 18, 2021, at 1:09 PM, Matt Burgess > wrote: > > > > > > +1 Release this package as nifi-1.13.2 > > > > > > Ran through release helper, verified fixes for NIFI-8297 and > > > NIFI-8334/8342 on an unsecured standalone instance. Thanks for RM'ing > > > and for the quick turnaround Joe! > > > > > > Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > > > Maven home: /Users/mburgess/.sdkman/candidates/maven/current > > > Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: > > > /Users/mburgess/.sdkman/candidates/java/8.0.282.hs-adpt/jre > > > Default locale: en_US, platform encoding: UTF-8 > > > OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac" > > > > > > On Thu, Mar 18, 2021 at 1:29 AM Joe Witt wrote: > > >> > > >> Hello, > > >> > > >> Please note this has a shorter than normal vote period as this > > >> corrects an important regression introduced in 1.13.1 and is purely a > > >> focused bug fix release. > > >> > > >> I am pleased to be calling this vote for the source release of Apache > > >> NiFi 1.13.2. > > >> > > >> The source zip, including signatures, digests, etc. can be found at: > > >> https://repository.apache.org/content/repositories/orgapachenifi-1180 > > >> > > >> The source being voted upon and the convenience binaries can be found > > at: > > >> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.2/ > > >> > > >> A helpful reminder on how the release candidate verification process > > works: > > >> > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > >> > > >> The Git tag is nifi-1.13.2-RC1 > > >> The Git commit ID is 174938e5e3bfe36951d5607d0d53e78604b0e07b > > >> > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=174938e5e3bfe36951d5607d0d53e78604b0e07b > > >> > > >> Checksums of nifi-1.13.2-source-release.zip: > > >> SHA256: > db18f57b4629367f21f70ec9a332294ec6b6c090a661c974b0fa7a80b09a79be > > >> SHA512: > > > 279c5701fd7b6d76206f5e79bdf1340d432a8e4834b8e786559095781c77c96a4594e1da040c6b404d1bab615acf5aa9dc60d289e460e6d7261f454f6210c66b > > >> > > >> Release artifacts are signed with the following key: > > >> https://people.apache.org/keys/committer/joewitt.asc > > >> > > >> KEYS file available here: > > >> https://dist.apache.org/repos/dist/release/nifi/KEYS > > >> > > >> 5 issues were closed/resolved for this release: > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350008 > > >> > > >> Release note highlights can be found here: > > >> > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.2 > > >> > > >> The vote will be open for 36 hours. > > >> Please download the release candidate and evaluate the necessary items > > >> including checking hashes, signatures, build from source, and test. > > >> Then please vote: > > >> > > >> [ ] +1 Release this package as nifi-1.13.2 > > >> [ ] +0 no opinion > > >> [ ] -1 Do not release this package because... > > > > >
Re: Apache Project issues with available GitHub Action resources across projects
This background is very helpful to keep in mind when evaluating new and updated unit tests. There are definitely some expensive tests that could be streamlined, but introducing a separate version lifecycle for framework and extensions seems like it is becoming more necessary. Moving to a Java 11 baseline would also reduce the need to build on multiple versions, but based on other discussions, it sounds like that is not currently scheduled. I have noticed that Windows builds can run for a longer period of time, is there a reason that the Windows workflow definition does not have the same timeout setting as the Linux and macOS builds? Introducing one would at least kill off problematic Windows builds. Regards, David Handermann On Mon, Apr 19, 2021 at 10:08 AM Joe Witt wrote: > Thanks for bringing this up. The most clear next step I can envision > at this point is that we break up our core framework from our > extensions. Not obvious how best to break this up but we need to. > The build times are insane. > > Joe > > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler > wrote: > > > > As you can probably imagine, it takes a lot of resources in order to CI > for all the Apache projects. Periodically this becomes an issue, as the > donated resources from cloud CI providers ( Travis and now GitHub Actions ) > end up queuing and delaying builds across Apache projects because of larger > projects and their requirements. > > > > The discussions center around a few common themes: > > > > - the CI requirements of large complex projects dominate the Apache Queue > > - how those projects can supplement their builds in a way acceptable to > ASF INFRA > > - how those projects can have per project metering such that a project > can pay for hours over the ’norm’ > > - how to optimize projects > > > > This issue is currently being discussed again on the @builds apache > list. I’m sending this over to the Nifi Dev community because Nifi itself > has been mentioned as one of the top users of GitHub action resources by > some measure. Many of the other projects are actively taking measures to > decrease or optimize their usage, and we should probably think about how we > can do so as well. > > > > Here is the *current* thread: > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3e > < > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3E > > > > > > Here is the message where 13 projects are listed > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8...@apache.org%3e > < > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8...@apache.org%3E > > > > > > There are many other threads regarding GitHub Action limits and > resources if you start scrolling back through the archives. > > > > I’m posting this to hopefully kick off some discussions. >
Re: Apache Project issues with available GitHub Action resources across projects
Chris,, Thanks for the description of the modular approach. Here is one GitHub action that supports conditional execution based on Git changes: https://github.com/dorny/paths-filter It would take some effort to implement an approach that covers all the bases, but using the Maven also-make and project-list command line options should support building both a module and its dependencies. The list of changes files can be passed to another action that could determine one or more modules to build. Regards, David Handermann On Mon, Apr 19, 2021 at 12:56 PM Chris Sampson wrote: > Our approach is the use a `git diff` between source and destination > branches when a branch is merged (and the same for on dev branch builds). > > Each component within the repo then checks for whether any files within its > directories were changed (bearing in mind a dev branch may consist of > multiple commits over time, so don't just look at changes in individual > commits). > > I'm not sure if the mechanics for the existing nifi repo though with its > use of: > > * GutHub Actions (can a git diff be performed at the start and then > following actions use the output?) > > * Maven with sub-modules (can they be optionally built based on such > conditions? The GitHub Action could create files depending upon what's been > changed in order to activate profiles maybe, would that work?) > > This is broadly how I've implemented a multi-component build previously, > but not using Maven sub-modules (each component is typically its own docker > image and built separately from others). > > > Cheers, > > Chris Sampson > > On Mon, 19 Apr 2021, 18:35 Joe Witt, wrote: > > > Chris, > > > > Yeah that would be very helpful. But do you have any idea how that > > might be achieved in this environment? > > > > Thanks > > > > On Mon, Apr 19, 2021 at 10:33 AM Chris Sampson > > wrote: > > > > > > Could an approach of building only the updated parts of the repo help > to > > > reduce build times? > > > > > > For example, changes to the classes under the AWS bundle (and only that > > > bundle) would only need those classes to be built and tested. > > > > > > Where such an approach gets a bit more complex is interdependence > between > > > parts of the repo. For example, if nifi-api is updated, should all > > classes > > > be built and tested? > > > > > > As part of a release, the entire repo could then be built and tested. > > > > > > This approach might help if the majority of repo changes are to > > individual > > > NARs rather than wider ranging. I'm also assuming this would be > possible > > > via GitHub Actions (I have no experience of them, but have implemented > > this > > > kind of approach in a Drone.io mono-repo previously). > > > > > > > > > Cheers, > > > > > > Chris Sampson > > > > > > On Mon, 19 Apr 2021, 17:16 David Handermann, < > exceptionfact...@gmail.com > > > > > > wrote: > > > > > > > This background is very helpful to keep in mind when evaluating new > and > > > > updated unit tests. There are definitely some expensive tests that > > could > > > > be streamlined, but introducing a separate version lifecycle for > > framework > > > > and extensions seems like it is becoming more necessary. Moving to a > > Java > > > > 11 baseline would also reduce the need to build on multiple versions, > > but > > > > based on other discussions, it sounds like that is not currently > > scheduled. > > > > > > > > I have noticed that Windows builds can run for a longer period of > > time, is > > > > there a reason that the Windows workflow definition does not have the > > same > > > > timeout setting as the Linux and macOS builds? Introducing one would > > at > > > > least kill off problematic Windows builds. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Mon, Apr 19, 2021 at 10:08 AM Joe Witt > wrote: > > > > > > > > > Thanks for bringing this up. The most clear next step I can > envision > > > > > at this point is that we break up our core framework from our > > > > > extensions. Not obvious how best to break this up but we need to. > > > > > The build times are insane. > > > > > > > > > > Joe > > > > > > >
Re: [ANNOUNCE] New Apache NiFi Committer Chris Sampson
Congratulations Chris! Looking forward to your continued contributions! Regards, David Handermann On Thu, May 13, 2021 at 3:38 PM Matt Burgess wrote: > Congratulations and welcome aboard Chris! > > On Thu, May 13, 2021 at 4:25 PM Joe Witt wrote: > > > > On behalf of the Apache NiFI PMC, I am very pleased to announce that > > Chris has accepted the PMC's invitation to become a committer on the > > Apache NiFi project. We greatly appreciate the contributions from Chris > > on the mailing list, PR reviews, code contributions, and especially rich > > communications in the Apache NiFi slack channels. > > > > Welcome and congratulations! >
Re: [discuss] nifi 1.14.0
Thanks for kicking off the discussion Joe! Of the many items that could be included in the next release, securing the default configuration as described in NIFI-8220 <https://issues.apache.org/jira/browse/NIFI-8220> would be great to have completed. Most of the elements are in place, and the current Pull Request for NIFI-8516 <https://github.com/apache/nifi/pull/5068> is under review. If there are any other achievable items that should be included as part of a secure default installation for NiFi, it would be helpful to add sub-tasks to NIFI-8220. The current scope is limited to a standalone installation, so issues regarding clustered deployments can be handled separately. If others are interested in evaluating the proposed new default configuration that requires HTTPS and leverages a generated username and password, feel free to provide feedback on NIFI-8516. Regards, David Handermann On Thu, May 27, 2021 at 6:51 PM Otto Fowler wrote: > I think NIFI-8625 and NIFI-8461 need to be understood and addressed. > > > > On May 27, 2021, at 13:29, Joe Witt wrote: > > > > Team, > > > > There has been a tremendous amount of work already on the 1.14 line as > shown: > > > > https://issues.apache.org/jira/projects/NIFI/versions/12349644 > > > > These include merging the nifi registry and minifi java into the nifi > > line itself. So when we release these things stay in sync and > > maintained. The release will now produce things like Apache NiFi, the > > Apache NiFi toolkit, Apache NiFi Registry, and Apache NiFi MiNiFi Java > > and the Apache NiFi stateless runtime as well. There have been many > > improvements to core nifi and stateless nifi now meaning we have the > > traditional execution form factor and this new stateless mode. We can > > now hot load nars from HDFS storage locations which could mean HDFS, > > blob storage in the cloud, etc.. There is a lot more. > > > > Anyway, I wanted to start circling the wagons for a 1.14 release. I'm > > happy to take on RM duties especially since there will be new elements > > to the release process. > > > > Thanks > >
Re: SSLPeerUnverifiedException following upgrade from 1.6.0 to 1.13.2
Hi Phil, Thanks for providing the stack trace. Recent versions of NiFi include updates to the OkHttp library, which modified the hostname verification process. OkHttp starting with version 3.10.0 made changes to TLS hostname verification, requiring that a certificate contain DNS Subject Alternative Names matching the connection hostname. Based on the error message, it appears that the certificates configured do not have any Subject Alternative Names, resulting in the SSLPeerUnverifiedException. Generating or obtaining new certificates that include the required DNS Subject Alternative Names should resolve the problem. Here's the release notes for OkHttp 3.10.0, referencing RFC 2818, which deprecated falling back to certificate common names for hostname verification: https://square.github.io/okhttp/changelog_3x/#version-3100 Regards, David Handermann On Thu, Jun 10, 2021 at 11:16 PM Phil H wrote: > Hi there, > > I upgraded an older dev setup today from 1.6.0 to 1.13.2. After a > couple of config tweaks, it’s “working”, but if I try and access the > interface at https://nifi2.domain.blah/ I get a message on screen > stating that nifi1.domain.blah is not verified. The logs contain this > same message, along with the stack trace. (This also happens if I > access nifi1 – it complains nifi2 is not verified). > > My keystore and truststore on both servers both contain the certs for > both servers, and the truststore additionally contains the CA that > signed both servers’ certificates. > > What am I missing? > > Thanks, > Phil > > > > > > 2021-06-11 23:51:20,970 WARN [Replicate Request Thread-1] > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request > GET /nifi-api/flow/current-user to nifi1.domain.blah:443 due to > javax.net.ssl.SSLPeerUnverifiedException: Hostname nifi1.domain.blah > not verified: > > certificate: sha256/Wv+eIBMlpsSS95xKF+Fry9C/jQhFbNS35yfJGK92/5U= > > DN: CN=nifi1.domain.blah, OU=domain, O=blah > > subjectAltNames: [] > > 2021-06-11 23:51:20,970 WARN [Replicate Request Thread-1] > o.a.n.c.c.h.r.ThreadPoolRequestReplicator > > javax.net.ssl.SSLPeerUnverifiedException: Hostname nifi1. > nifi1.domain.blah not verified: > > certificate: sha256/Wv+eIBMlpsSS95xKF+Fry9C/jQhFbNS35yfJGK92/5U= > > DN: CN=nifi1.domain.blah OU=domain, O=blah > > subjectAltNames: [] > > at > > okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:389) > > at > > okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337) > > at > okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209) > > at > > okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) > > at > > okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) > > at > okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) > > at > okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) > > at > > okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) > > at > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) > > at > okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) > > at > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) > > at > okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) > > at > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) > > at > > okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) > > at > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) > > at > > okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) > > at > okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) > > at > > org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:132) > > at > > org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:126) > > at > > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestR
Re: [discuss] nifi 1.14.0
Joe, Thanks for following up. The PR for NIFI-8516 has gone through several rounds of feedback, I believe it is about ready to go, pending confirmation that the ability to set custom credentials addresses the ease of use concern. Regards, David Handermann On Fri, Jun 11, 2021 at 11:41 AM Joe Witt wrote: > David, > > Ok thanks - do you have a sense of when what you see as good 1.14 > specific work will be merged? Do you have the reviewers/engagement > you need? > > This 1.14 is already pretty packed but definitely agree we need to > make real progress on secure by default and this release is a great > time to take the first big step. > > Thanks > > On Mon, May 31, 2021 at 5:52 AM David Handermann > wrote: > > > > Thanks for kicking off the discussion Joe! > > > > Of the many items that could be included in the next release, securing > the > > default configuration as described in NIFI-8220 > > <https://issues.apache.org/jira/browse/NIFI-8220> would be great to have > > completed. Most of the elements are in place, and the current Pull > Request > > for NIFI-8516 <https://github.com/apache/nifi/pull/5068> is under > review. > > If there are any other achievable items that should be included as part > of > > a secure default installation for NiFi, it would be helpful to add > > sub-tasks to NIFI-8220. The current scope is limited to a standalone > > installation, so issues regarding clustered deployments can be handled > > separately. If others are interested in evaluating the proposed new > > default configuration that requires HTTPS and leverages a generated > > username and password, feel free to provide feedback on NIFI-8516. > > > > Regards, > > David Handermann > > > > On Thu, May 27, 2021 at 6:51 PM Otto Fowler > wrote: > > > > > I think NIFI-8625 and NIFI-8461 need to be understood and addressed. > > > > > > > > > > On May 27, 2021, at 13:29, Joe Witt wrote: > > > > > > > > Team, > > > > > > > > There has been a tremendous amount of work already on the 1.14 line > as > > > shown: > > > > > > > > https://issues.apache.org/jira/projects/NIFI/versions/12349644 > > > > > > > > These include merging the nifi registry and minifi java into the nifi > > > > line itself. So when we release these things stay in sync and > > > > maintained. The release will now produce things like Apache NiFi, > the > > > > Apache NiFi toolkit, Apache NiFi Registry, and Apache NiFi MiNiFi > Java > > > > and the Apache NiFi stateless runtime as well. There have been many > > > > improvements to core nifi and stateless nifi now meaning we have the > > > > traditional execution form factor and this new stateless mode. We > can > > > > now hot load nars from HDFS storage locations which could mean HDFS, > > > > blob storage in the cloud, etc.. There is a lot more. > > > > > > > > Anyway, I wanted to start circling the wagons for a 1.14 release. > I'm > > > > happy to take on RM duties especially since there will be new > elements > > > > to the release process. > > > > > > > > Thanks > > > > > > >
Re: [discuss] nifi 1.14.0
Thanks to Mark Payne, NIFI-8516 is now merged, so that covers current open issues around securing the default configuration. Regards, David Handermann On Fri, Jun 11, 2021 at 11:55 AM David Handermann < exceptionfact...@apache.org> wrote: > Joe, > > Thanks for following up. The PR for NIFI-8516 has gone through several > rounds of feedback, I believe it is about ready to go, pending confirmation > that the ability to set custom credentials addresses the ease of use > concern. > > Regards, > David Handermann > > On Fri, Jun 11, 2021 at 11:41 AM Joe Witt wrote: > >> David, >> >> Ok thanks - do you have a sense of when what you see as good 1.14 >> specific work will be merged? Do you have the reviewers/engagement >> you need? >> >> This 1.14 is already pretty packed but definitely agree we need to >> make real progress on secure by default and this release is a great >> time to take the first big step. >> >> Thanks >> >> On Mon, May 31, 2021 at 5:52 AM David Handermann >> wrote: >> > >> > Thanks for kicking off the discussion Joe! >> > >> > Of the many items that could be included in the next release, securing >> the >> > default configuration as described in NIFI-8220 >> > <https://issues.apache.org/jira/browse/NIFI-8220> would be great to >> have >> > completed. Most of the elements are in place, and the current Pull >> Request >> > for NIFI-8516 <https://github.com/apache/nifi/pull/5068> is under >> review. >> > If there are any other achievable items that should be included as part >> of >> > a secure default installation for NiFi, it would be helpful to add >> > sub-tasks to NIFI-8220. The current scope is limited to a standalone >> > installation, so issues regarding clustered deployments can be handled >> > separately. If others are interested in evaluating the proposed new >> > default configuration that requires HTTPS and leverages a generated >> > username and password, feel free to provide feedback on NIFI-8516. >> > >> > Regards, >> > David Handermann >> > >> > On Thu, May 27, 2021 at 6:51 PM Otto Fowler >> wrote: >> > >> > > I think NIFI-8625 and NIFI-8461 need to be understood and addressed. >> > > >> > > >> > > > On May 27, 2021, at 13:29, Joe Witt wrote: >> > > > >> > > > Team, >> > > > >> > > > There has been a tremendous amount of work already on the 1.14 line >> as >> > > shown: >> > > > >> > > > https://issues.apache.org/jira/projects/NIFI/versions/12349644 >> > > > >> > > > These include merging the nifi registry and minifi java into the >> nifi >> > > > line itself. So when we release these things stay in sync and >> > > > maintained. The release will now produce things like Apache NiFi, >> the >> > > > Apache NiFi toolkit, Apache NiFi Registry, and Apache NiFi MiNiFi >> Java >> > > > and the Apache NiFi stateless runtime as well. There have been many >> > > > improvements to core nifi and stateless nifi now meaning we have the >> > > > traditional execution form factor and this new stateless mode. We >> can >> > > > now hot load nars from HDFS storage locations which could mean HDFS, >> > > > blob storage in the cloud, etc.. There is a lot more. >> > > > >> > > > Anyway, I wanted to start circling the wagons for a 1.14 release. >> I'm >> > > > happy to take on RM duties especially since there will be new >> elements >> > > > to the release process. >> > > > >> > > > Thanks >> > > >> > > >> >
Re: nifi windows prpblem
Thanks for providing the error message, can you include a few additional details about your system configuration? In particular, which version of Java are you using? This sounds like the following issue, which is fixed in the current main branch and will be including in the next release of NiFi: https://issues.apache.org/jira/browse/NIFI-6714 Regards, David Handermann On Sat, Jun 12, 2021 at 10:31 AM MaHmOuD El-Sayed wrote: > .bootstrap.config.log.dir=C:\Users\MaHmOuD\DOWNLO~1\COMPRE~1\NIFI-1~1.2\bin\..\\logs > org.apache.nifi.NiFi > Exception in thread "main" java.lang.reflect.InaccessibleObjectException: > Unable to make public long java.lang.ProcessImpl.pid() accessible: module > java.base does not "opens java.lang" to unnamed module @ba8d91c >
Re: [VOTE] Release Apache NiFi 1.14.0 (rc2)
+1 (non-binding) - Built from source on Ubuntu 21.04 with Azul Zulu 11.0.10 - Ran NiFi on Azul Zulu 11.0.10 and 1.8.0.282 - Configured encrypted Flow File Repository using KeyStore Key Provider and PKCS12 - Configured encrypted Flow File Swap Manager - Tested set-sensitive-properties-key and set-single-user-credentials commands - Tested flow with PutTCP and ListenTCP with TLS 1.3 - Tested flow with UnpackContent using Zip file with bzip2 compression - Tested flow with EncryptContentPGP and DecryptContentPGP - Tested NiFi Toolkit using AES-GCM encryption of nifi.properties including new repository password property - Tested NiFi Registry with bucket creation and NiFi process group version control - Test NiFi Stateless with GetFile and UnpackContent flow downloaded from NiFi process group Regards, David Handermann On Mon, Jul 12, 2021 at 6:31 AM Arpad Boda wrote: > +1 (binding) > > Built on Debian 10 using Java 11, started, designed a simple flow. > Verified hashes, signatures. > > Ps.: some tests failed, so only built without them. > > On Mon, Jul 12, 2021 at 12:38 PM Joe Gresock wrote: > > > +1 (non-binding) > > > > Verified signature and hashes > > Full build with contrib check with OpenJDK Runtime Environment (Zulu > > 8.54.0.21-CA-macosx) (build 1.8.0_292-b10) > > Upgraded a single secure nifi 1.13.2 cluster with an existing flow, which > > started successfully. > > Verified TLS 1.3 works using an existing flow with > > HandleHttpRequest/Response and InvokeHttp > > Restarted with nifi.security.autoreload.enabled=true and verified that I > > was able to replace a keystore and truststore while NiFi was running > > Upgraded a 3-node secure nifi 1.13.2 cluster with an existing flow, which > > started successfully. > > Verified that load balanced connections work as expected, before and > after > > offloading a node. > > Used the encrypt-config.sh tool from Toolkit to encrypt nifi.properties > and > > nifi-registry.properties using AES/GCM, verified both apps started > > successfully. > > Used encrypt-config.sh to migrate the encryption to the > > HASHICORP_VAULT_TRANSIT protection scheme, verified both apps started > > successfully. > > Built a DataFlow and saved to file, then ran it via Stateless NiFi, > using a > > custom ParamaterProvider, verified that parameters were correctly > provided. > > > > On Sun, Jul 11, 2021 at 1:33 PM Robert Fellows > > wrote: > > > > > +1 (non-binding) > > > > > > Verified signature and hashes > > > Full build with contrib check with OpenJDK Runtime Environment > > AdoptOpenJDK > > > (build 11.0.8+10) > > > Started new instance, verified it was secured by default. > > > Changed the username and password using bin/nifi.sh > > > set-single-user-credentials > > > Logged in with the new user/password i created > > > Fired up registry, created a bucket > > > Connected nifi to registry, versioned a flow. Imported a flow into > > another > > > process group. > > > General App usage (Firefox 89.0.2) > > > > > > Thanks for RM'ing this Joe! > > > > > > -- Rob Fellows > > > > > > On Sat, Jul 10, 2021 at 9:31 PM Mark Payne > wrote: > > > > > > > +1 (binding) > > > > > > > > Verified the hashes. > > > > Performed full build with contrib-check profile > > > > Ran all system tests > > > > Started the newly created instance, verified that it was secure by > > > default. > > > > Changed username & password and verified the behavior. > > > > Created a 2-node cluster, secured by mutual TLS and verified > behavior. > > > > > > > > Started nifi registry. Created a bucket. > > > > Pushed to the bucket from standalone instance. Then added a second > > > version > > > > of flow. > > > > Downloaded flow onto cluster and switched between versioned a few > > times. > > > > > > > > Verified behavior of new default backpressure thresholds. > > > > Built a DataFlow and saved to file, then ran it via Stateless NiFi > > using > > > > command-line parameter overrides. > > > > Started DataFlow using Stateless nifi and the kafka connector. > Verified > > > > this behavior. > > > > > > > > Encountered no issues this time. > > > > > > > > Thanks for putting the RC together, Joe! > > > > > > > > > On Jul 10, 2021, at 6:40 PM, Joe Witt wrote: > > > > > >
Re: [discuss] nifi 1.14.0
Chad, Can you provide the output of the test failures, specifically which unit tests failed and any associated error messages? That would help narrow down the potential source of the problem. Regards, David Handermann On Mon, Jul 12, 2021 at 9:05 PM Chad Zobrisky wrote: > Anyone else having issues building the RC? I'm also having the same test > failure on main. I can build if I skip tests. > > Ubuntu 20.04 > java OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 > maven 3.6.3 > > It fails on tests, specifically nifi-database-utils. I've cleared the maven > repository, made sure my java is up to date and tried multiple times with > the same result. I had a different test failure with some slf4j issue on a > slightly older java 11 release from April, and updated to resolve that > issue. > > Thanks, > Chad > > On Mon, Jul 12, 2021 at 8:20 PM Otto Fowler > wrote: > > > I’ll take care of it. I’ll make the minimum 3.6.0 > > > > From: Joe Witt > > Reply: dev@nifi.apache.org > > Date: July 12, 2021 at 18:25:28 > > To: dev@nifi.apache.org > > Subject: Re: [discuss] nifi 1.14.0 > > > > Yeah we just need to update the latest required version . We're going > > all in on all the HTTPS requirements and such so we're probably really > > tight on required version at this point anyway. > > > > I'd not be sinking the RC for this. Just a good JIRA to work on. > > > > Thanks > > > > On Mon, Jul 12, 2021 at 12:59 PM Otto Fowler > > wrote: > > > > > > NIFI-8778 > > > > > > From: Otto Fowler > > > Reply: Otto Fowler > > > Date: July 12, 2021 at 13:56:36 > > > To: dev@nifi.apache.org > > > Subject: Re: [discuss] nifi 1.14.0 > > > > > > I see an issue, though I’m not sure if we want to cancel the RC so I’m > > > asking before voting. > > > The documentation states that we require Apache Maven 3.1.1 or newer, > but > > > the build fails with the error: > > > > > > [ERROR] Failed to execute goal > > > com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm > > > (install-node-and-npm) on project nifi-web-ui: The plugin > > > com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version > > > 3.6.0 -> [Help 1] > > > > > > I have maven version: > > > > > > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > > > 2018-06-17T14:33:14-04:00) > > > Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec > > > Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: > > > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > > > > > > So, the documentation on building nifi is wrong. > > > > > > > > > > > > From: Joe Witt > > > Reply: dev@nifi.apache.org > > > Date: July 9, 2021 at 10:24:58 > > > To: dev@nifi.apache.org > > > Subject: Re: [discuss] nifi 1.14.0 > > > > > > Still working on the 1.14 RC. Has been a series of issues with the > > > build. Will be up soon hopefully! > > > > > > On Thu, Jul 8, 2021 at 1:41 PM Mark Payne > wrote: > > > > > > > > Joe, > > > > > > > > I just filed a BUG Jira [1]. It’s a bit of a corner case, but when it > > > occurs it can cause a pretty big problem. Should have a fix up very > > > shortly. Will leave it up to you whether or not you think we should get > > > this into 1.14.0. > > > > > > > > Thanks > > > > -Mark > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/NIFI-8771 > > > > > > > > > > > > On Jul 8, 2021, at 10:26 AM, Joe Witt > > joe.w...@gmail.com>> wrote: > > > > > > > > Team, > > > > > > > > The 1.14 RC1 build is underway. Hopefully I will email artifacts > > > > today. Reminder this should now generate convenience binaries for > > > > nifi, stateless nifi, minifi java, nifi registry and the associated > > > > toolkits all in a single release process which also keeps these > things > > > > in sync. > > > > > > > > Please do not tag any further fix versions to 1.14.0. Use 1.15.0 from > > > > here. I'll pull things in if RCs fail/etc. > > > > > > > > Thanks > > > > > > > > On Tue, Jul 6, 2021 at 8:21 AM Joe Witt > > joe.w...@gmail.com>>
Re: [DISCUSS] NiFi Registry -> NiFi consensus
Thanks for handling this, Matt. As one of the reviewers on the PR, I vote +1 (non-binding). Regards, David Handermann On Fri, Jul 16, 2021 at 1:20 PM Matt Burgess wrote: > All, > > We've been asked to record our consensus for the move of NiFi Registry > to the NiFi codebase in a mailing list thread for posterity. Most > discussion happened on the PR but INFRA would like a link to this > thread showing consensus from PMC members, committers, and the > community. > > I didn't put my +1 on the PR but I am in favor of moving the NiFi > Registry codebase into NiFi :) Please feel free to share your thoughts > as well. > > Thank you, > Matt >
[DISCUSS] NiFi 2.0 Release Goals
Team, With all of the excellent work that many have contributed to NiFi over the years, the code base has also accumulated some amount of technical debt. A handful of components have been marked as deprecated, and some components remain in the code base to support integration with old versions of various products. Following the principles of semantic versioning, introducing a major release would provide the opportunity to remove these deprecated and unsupported components. Rather than focusing the next major release on new features, what do you think about focusing on technical debt removal? This approach would not make for the most interesting release, but it provides the opportunity to clean up elements that involve breaking changes. Focusing on technical debt, at least three primary goals come to mind for the next major release: 1. Removal of deprecated and unmaintained components 2. Require Java 11 as the minimum supported version 3. Transition internal date and time handling to JSR 310 java.time components *Removing Deprecated Components* Removing support for older and deprecated components provides a great opportunity to improve the overall security posture when it comes to maintaining dependencies. The OWASP dependency plugin report currently generates 50 MB of HTML for questionable dependencies, many of which are related to old versions of various libraries. As a starting point, here are a handful of components and extension modules that could be targeted for removal in a major version: - PostHTTP and GetHTTP - ListenLumberjack and the entire nifi-lumberjack-bundle - ListenBeats and the entire nifi-beats-bundle - Elasticsearch 5 components - Hive 1 and 2 components *Requiring Java 11* Java 8 is now over seven years old, and NiFi has supported general compatibility with Java 11 for several years. NiFi 1.14.0 incorporated internal improvements specifically related to TLS 1.3, which allowed closing out the long-running Java 11 compatibility epic NIFI-5174. Making Java 11 the minimum required version provides the opportunity to address any lingering edge cases and put NiFi in a better position to support current Java versions. *JSR 310 for Date and Time Handling* Without making the scope too broad, transitioning internal date and time handling to use DateTimeFormatter instead of SimpleDateFormat would provide a number of advantages. The Java Time components provide much better clarity when it comes to handling localized date and time representations, and also avoid the inherent confusion of java.sql.Date extending java.util.Date. Many internal components, specifically Record-oriented processors and services, rely on date parsing, leading to confusion and various workarounds. The pattern formats of SimpleDateFormat and DateTimeFormatter are very similar, but there are a few subtle differences. Making this transition would provide a much better foundation going forward. *Conclusion* Thanks for giving this proposal some consideration. Many of you have been developing NiFi for years and I look forward to your feedback. I would be glad to put together a more formalized recommendation on Confluence and write up Jira epics if this general approach sounds agreeable to the community. Regards, David Handermann
Re: [DISCUSS] NiFi 2.0 Release Goals
Thanks to everyone who has provided feedback thus far, it is good to see general interest in moving forward with a release focusing on technical debt removal. It seems like it would be helpful to start gathering removal candidates on a Confluence wiki page, and then turning those into Jira issues. I am open to suggestions here, but it seems like creating several epics related to high level goals would be a good way to organize the work. I can start with creating a Confluence wiki page to capture some of these initial details. A release version 2.0.0 already exists in Jira, so that could be used for tagging planned issues. >From a branching and maintenance perspective, perhaps PMC members could describe how this should work? Should the main branch become the branch planned for 2.0.0 work, and does that require a vote for approval to proceed? Or should a new 2.0 branch be created and new commits applied selectively? Regards, David Handermann On Fri, Jul 23, 2021 at 11:29 AM Matt Burgess wrote: > Along with the itemized list for ancient components we should look at > updating versions of drivers, SDKs, etc. for external systems such as > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0 > is probably the right time to get things up to date to make them more > useful to more people. > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough wrote: > > > > I'm a +1 for removing pretty much all of this stuff. There are security > > implications to keeping old dependencies around, so the more old code we > > can remove the better. I agree that eventually we need to move to > > supporting only Java 11+, and as our next release will probably be about > 4 > > - 6 months from now that doesn't seem too soon. We could potentially > break > > this in two and remove the deprecated processors and leave 1.x on Java 8, > > and finally start on 2.x which would support only Java 11. I'm unsure of > > what implications changing the date and time handling would have - for > > running systems that use long term historical logs, unexpected impacts to > > time logging could be a problem. > > > > As Joe says I think feature work will have to be dedicated to 2.x and we > > could support 1.x for security fixes for some period of time. 2.x seems > > like a gargantuan task but it's probably time to get started. Not sure > how > > we handle all open PRs and the transition between 1.x and 2.x. > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt wrote: > > > > > Jon > > > > > > You're right we have to be careful and you're right there are still > > > significant Java 8 users out there. But we also have to be careful > > > about security and sustainability of the codebase. If we had talked > > > about this last year when that article came out I'd have agreed it is > > > too early. Interestingly that link seems to get updated and I tried > > > [1] and found more recent data (not sure how recent). Anyway it > > > suggests Java 8 is still the top dog but we see good growth on 11. In > > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem > > > to care about Java 11 until later half last year and now suddenly it > > > is all over the place. > > > > > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in > > > work on the 1.x line just being blunt. We did this many years ago > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or > > > so) but it was purely bug fix/security related bits. We would need to > > > do something similar. But feature work would almost certainly go to > > > the 2.x line. Maybe there are other workable models but my instinct > > > suggests this is likely to follow a similar path. > > > > > > ...anyway I agree it isn't that easy of a call to dump Java 8. We > > > need to make the call in both the interests of the user base and the > > > contributor base of the community. > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/ > > > > > > > > > Thanks > > > Joe > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt wrote: > > > > > > > > Russ > > > > > > > > Yeah the flow registry is a key part of it. But also now you can > > > > download the flow definition in JSON (upload i think is there now > > > > too). Templates offered a series of challenges such as we store them > > > > in the flow definition which has made flows massive in an unintended &
Re: [DISCUSS] NiFi 2.0 Release Goals
Team, I created the following page on the Apache NiFi wiki to track proposed goals and particular focus areas. If the page should go under another section of the wiki, it can be moved. https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Proposed+Release+Goals The Primary Goals section is intended to cover high-level categories, and each subsection includes some initial elements that seem to fit in that category. This initial version is not intended to be exhaustive or definitive, but should keep this conversation moving forward. I can make updates based on feedback in this thread, and others with access are welcome to make direct changes. As a list of proposed goals, the purpose is not to exclude additional ideas, but to promote a focused discussion toward the next major release. I would be glad to help write up Jira issues when we get to that point. For now, more feedback is better in terms of what should be covered under the general heading of technical debt reduction. Regards, David Handermann On Fri, Jul 23, 2021 at 12:11 PM Russell Bateman wrote: > Bringing up Elastic also reminds me that the Elastic framework has just > recently transitioned out of Open Source, so to acknowledge that, maybe > some effort toward OpenSearch--I say this not understanding exactly how > this sort of thing is considered in a large-scale, world-class software > project like Apache NiFi. (I'm not a contributor, just a grateful > consumer.) > > Russ > > On 7/23/21 10:28 AM, Matt Burgess wrote: > > Along with the itemized list for ancient components we should look at > > updating versions of drivers, SDKs, etc. for external systems such as > > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0 > > is probably the right time to get things up to date to make them more > > useful to more people. > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough > wrote: > >> I'm a +1 for removing pretty much all of this stuff. There are security > >> implications to keeping old dependencies around, so the more old code we > >> can remove the better. I agree that eventually we need to move to > >> supporting only Java 11+, and as our next release will probably be > about 4 > >> - 6 months from now that doesn't seem too soon. We could potentially > break > >> this in two and remove the deprecated processors and leave 1.x on Java > 8, > >> and finally start on 2.x which would support only Java 11. I'm unsure of > >> what implications changing the date and time handling would have - for > >> running systems that use long term historical logs, unexpected impacts > to > >> time logging could be a problem. > >> > >> As Joe says I think feature work will have to be dedicated to 2.x and we > >> could support 1.x for security fixes for some period of time. 2.x seems > >> like a gargantuan task but it's probably time to get started. Not sure > how > >> we handle all open PRs and the transition between 1.x and 2.x. > >> > >> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt wrote: > >> > >>> Jon > >>> > >>> You're right we have to be careful and you're right there are still > >>> significant Java 8 users out there. But we also have to be careful > >>> about security and sustainability of the codebase. If we had talked > >>> about this last year when that article came out I'd have agreed it is > >>> too early. Interestingly that link seems to get updated and I tried > >>> [1] and found more recent data (not sure how recent). Anyway it > >>> suggests Java 8 is still the top dog but we see good growth on 11. In > >>> my $dayjob this aligns to what I'm seeing too. Customers didn't seem > >>> to care about Java 11 until later half last year and now suddenly it > >>> is all over the place. > >>> > >>> I think once we put out a NiFi 2.0 release we'd see rapid decrease in > >>> work on the 1.x line just being blunt. We did this many years ago > >>> with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or > >>> so) but it was purely bug fix/security related bits. We would need to > >>> do something similar. But feature work would almost certainly go to > >>> the 2.x line. Maybe there are other workable models but my instinct > >>> suggests this is likely to follow a similar path. > >>> > >>> ...anyway I agree it isn't that easy of a call to dump Java 8. We > >>> need to make the call in both the interests of the user base and the &
Re: [DISCUSS] NiFi 2.0 Release Goals
Thanks for pointing out the standard NAR bundles, Mark. There are a number of components in the standard NAR bundles with particular dependencies that would make more sense in separate NARs. Reorganizing the standard NAR to components with limited dependencies and wide applicability would definitely help with future maintenance. Regards, David Handermann On Sat, Jul 24, 2021 at 10:57 AM Mark Payne wrote: > There’s also some code that exists in order to maintain backward > compatibility in the repositories. I would very much like the repositories > to contain no unnecessary code. And swap file format supports really old > formats. And the old impls of the repositories themselves, like > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff there > that can be removed. And some methods in ProcessSession that are never used > by any processor in the codebase but exists in the public API so can’t be > removed till 2.0. > > I think his is also a great time to clean up the “standard nar.” At this > point, it’s something like 70 MB. And many of the components there are not > really “standard” - things like connecting to FTP & SFTP servers, XML > processing, Jolt transform, etc. could potentially be moved into other > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB is not > necessary for stateless or minifi java. Lots of things probably to > reconsider within the standard nar. > > I definitely think this is a reasonable approach, to allow for a 2.0 that > is not a huge feature release but allows the project to be simpler and more > nimble. > > Thanks > -Mark > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen mikerthom...@gmail.com>> wrote: > > Russell, > > AFAICT from looking at Elastic's repos, the low level REST client is > still fine. > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > Our Elasticsearch support is spread over two NARs at present. One uses > OkHttp and the other uses that low level Elastic REST client. > Therefore, I think we're fine on licensing for the moment. > > Mike > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <mailto:r...@windofkeltia.com>> wrote: > > Bringing up Elastic also reminds me that the Elastic framework has just > recently transitioned out of Open Source, so to acknowledge that, maybe > some effort toward OpenSearch--I say this not understanding exactly how > this sort of thing is considered in a large-scale, world-class software > project like Apache NiFi. (I'm not a contributor, just a grateful > consumer.) > > Russ > > On 7/23/21 10:28 AM, Matt Burgess wrote: > Along with the itemized list for ancient components we should look at > updating versions of drivers, SDKs, etc. for external systems such as > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0 > is probably the right time to get things up to date to make them more > useful to more people. > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough thena...@gmail.com>> wrote: > I'm a +1 for removing pretty much all of this stuff. There are security > implications to keeping old dependencies around, so the more old code we > can remove the better. I agree that eventually we need to move to > supporting only Java 11+, and as our next release will probably be about 4 > - 6 months from now that doesn't seem too soon. We could potentially break > this in two and remove the deprecated processors and leave 1.x on Java 8, > and finally start on 2.x which would support only Java 11. I'm unsure of > what implications changing the date and time handling would have - for > running systems that use long term historical logs, unexpected impacts to > time logging could be a problem. > > As Joe says I think feature work will have to be dedicated to 2.x and we > could support 1.x for security fixes for some period of time. 2.x seems > like a gargantuan task but it's probably time to get started. Not sure how > we handle all open PRs and the transition between 1.x and 2.x. > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt joe.w...@gmail.com>> wrote: > > Jon > > You're right we have to be careful and you're right there are still > significant Java 8 users out there. But we also have to be careful > about security and sustainability of the codebase. If we had talked > about this last year when that article came out I'd have agreed it is > too early. Interestingly that link seems to get updated and I tried > [1] and found more recent data (not sure how recent). Anyway it > suggests Java 8 is still the top dog but we see good growth on 11. In &g
Re: [DISCUSS] NiFi 2.0 Release Goals
Chris, Thanks for the reply and recommendations. It seems like some of the work to reorganize the module structure could be done outside of a major release, but it would be great to target any breaking changes for 2.0. Perhaps a separate feature proposal on module restructuring, with the goal of supporting optimized builds, would be a helpful way to move that part of the discussion forward. Regarding updating AWS SDK to version 2, it seems like that might be possible now. I haven't taken a close look at the referencing components, so I'm not sure about the level of effort involved. Minor NiFi version updates have incorporated major new versions of dependencies. For example, NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one hand, including the AWS SDK update as part of a major release seems helpful, but unless there are changes that break existing component properties, upgrading the AWS SDK could be worked independently. Others may have more insight into particular usage of that library. Regards, David Handermann On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson wrote: > Might be worth considering refactoring the build as part of this work too, > e.g. only building the bits of the repo affected by a commit, etc. - > discussed briefly in previous threads but don't think any changes made yet. > If NARs/components are likely to be split up and refactored then such work > around the build probably makes sense to consider. > > I've a couple of PRs open that include updates to Elasticsearch versions > already, although I stopped at 7.10.2 (after which Elastic changed licence) > in case there were licence concerns. But more work can be done to tidy up > the processors, absolutely. > > AWS libraries to v2 would seem a sensible move and a refactor of those > processors as well. > > > Cheers, > > Chris Sampson > > On Sat, 24 Jul 2021, 17:47 David Handermann, > wrote: > > > Thanks for pointing out the standard NAR bundles, Mark. There are a > number > > of components in the standard NAR bundles with particular dependencies > that > > would make more sense in separate NARs. Reorganizing the standard NAR to > > components with limited dependencies and wide applicability would > > definitely help with future maintenance. > > > > Regards, > > David Handermann > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne > wrote: > > > > > There’s also some code that exists in order to maintain backward > > > compatibility in the repositories. I would very much like the > > repositories > > > to contain no unnecessary code. And swap file format supports really > old > > > formats. And the old impls of the repositories themselves, like > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff > > there > > > that can be removed. And some methods in ProcessSession that are never > > used > > > by any processor in the codebase but exists in the public API so can’t > be > > > removed till 2.0. > > > > > > I think his is also a great time to clean up the “standard nar.” At > this > > > point, it’s something like 70 MB. And many of the components there are > > not > > > really “standard” - things like connecting to FTP & SFTP servers, XML > > > processing, Jolt transform, etc. could potentially be moved into other > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB is > > not > > > necessary for stateless or minifi java. Lots of things probably to > > > reconsider within the standard nar. > > > > > > I definitely think this is a reasonable approach, to allow for a 2.0 > that > > > is not a huge feature release but allows the project to be simpler and > > more > > > nimble. > > > > > > Thanks > > > -Mark > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen > > > mikerthom...@gmail.com>> wrote: > > > > > > Russell, > > > > > > AFAICT from looking at Elastic's repos, the low level REST client is > > > still fine. > > > > > > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > > > > > Our Elasticsearch support is spread over two NARs at present. One uses > > > OkHttp and the other uses that low level Elastic REST client. > > > Therefore, I think we're fine on licensing for the moment. > > > > > > Mike > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman > > &l
Re: [DISCUSS] NiFi 2.0 Release Goals
Mark, Thanks for the feedback. It may be better to start a separate thread on PostHTTP, but can you provide an example flow demonstrating the performance differences between PostHTTP and InvokeHTTP? PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP uses OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for OkHttp, so it would be important to test with the most recent version. It is also worth noting that test classes for PostHTTP are now integration tests as opposed to unit tests, which means they are not executed as part of the automated builds. The NiFi documentation includes the contents of the DeprecationNotice for PostHTTP: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html Regards, David Handermann On Tue, Jul 27, 2021 at 9:56 AM Mark Bean wrote: > I'm strongly in favor of reducing tech debt, and moving deliberately to a > 2.0 release. I have a concern with just one processor that is currently > marked as deprecated: PostHTTP. (I have not evaluated specifically any > other deprecated components; I cannot say if there are or are not similar > issues elsewhere.) I understand the rationale for deprecating this > processor in that it eliminates a processor whose functionality is > available elsewhere, specifically in the more flexible InvokeHTTP. However, > in my experience and testing, PostHTTP performs transfers with far greater > throughput than InvokeHTTP. I would not be in favor of removing PostHTTP > unless/until InvokeHTTP is refactored to increase its throughput > capability. > > Has anyone else continued to use PostHTTP over InvokeHTTP for similar > reasons? Or, is there a performance-related configuration for InvokeHTTP I > may have missed? > > Also, in order to help facilitate a smooth transition to 2.0 from a user > perspective, would it be advisable to add some sort of visual indicator in > the UI for components that are currently annotated as @Deprecated? This > might help users proactively modify their flow prior to a release that > would otherwise break it. > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende wrote: > > > With the merging of MiNiFi and registry into the NiFi repo, we've > > actually gone more towards a mono-repo where everything is released > > together. Eventually it would still be nice to have a smaller base > > distribution containing just the framework and standard NARs, but it > > is hard to tackle that until we provide a way for users to easily > > obtain the NARs in some other way. > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes > > wrote: > > > > > > Given the major version number shift and the spliting up of processors > > into > > > multiple NAR's. I'd like to suggest that we start individually > versioning > > > individual NARs/ bundles. > > > > > > I can see this bringing a large number of benifits including making > Nifi > > > more deployable with things RPM, but also potentially allowing those > that > > > have to deploy managed Nifi instances easier to upgrade. > > > > > > Edward > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, > wrote: > > > > > > > The issue with updating the aws sdk is if it breaks any one of the > > > > processors. > > > > the Web Gateway API invoke processor for example is not using a high > > level > > > > purpose build client and may break. > > > > > > > > If we change the aws version, we need to coordinate in such a way > that > > they > > > > all > > > > can come along reasonably. > > > > IE: what happens if 1 or 2 break but the rest or OK? > > > > > > > > > > > > > > > > From: David Handermann > > > > > > > > Reply: dev@nifi.apache.org < > dev@nifi.apache.org> > > > > Date: July 26, 2021 at 09:33:42 > > > > To: dev@nifi.apache.org > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > > > Chris, > > > > > > > > Thanks for the reply and recommendations. It seems like some of the > > work to > > > > reorganize the module structure could be done outside of a major > > release, > > > > but it would be great to target any breaking changes for 2.0. > Perhaps a > > > > separate feature proposal on module restructuring, with the goal of > > > > supporting optimized builds, would be a helpful way to move that part > > of > > >
Re: [DISCUSS] NiFi 2.0 Release Goals
Thanks Mark, providing a template or comparison statistics with Java versions and component configuration details would be very helpful. If it is possible to run tests using a public API or deployable service, that would also help confirm potential differences. Showing a deprecation notice in the UI could be helpful, perhaps as a configurable option. NIFI-8650 describes a general Flow Analysis capability, and it seems like that might be one possible way to surface deprecation warnings. For something more specific to component deprecation, it seems important to find a balance between making it obvious and making it something that ends up getting ignored. Regards, David Handermann On Tue, Jul 27, 2021 at 10:28 AM Mark Bean wrote: > I'll start a new thread for PostHTTP when I get a template and/or detailed > stats. > > I know the deprecation is noted in the documentation. That's a necessary > and minimum level of notification. I was suggesting it be more obvious in > the UI. I think it would be beneficial to somehow be aware of the > deprecation status simply by looking at the flow (perhaps even on the > summary pages too), and without having to open the documentation for every > processor to confirm whether or not a component is marked as deprecated. > > Thanks, > Mark > > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann < > exceptionfact...@apache.org> wrote: > > > Mark, > > > > Thanks for the feedback. It may be better to start a separate thread on > > PostHTTP, but can you provide an example flow demonstrating the > performance > > differences between PostHTTP and InvokeHTTP? > > > > PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP uses > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for OkHttp, > > so it would be important to test with the most recent version. It is also > > worth noting that test classes for PostHTTP are now integration tests as > > opposed to unit tests, which means they are not executed as part of the > > automated builds. > > > > The NiFi documentation includes the contents of the DeprecationNotice for > > PostHTTP: > > > > > > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html > > > > Regards, > > David Handermann > > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean wrote: > > > > > I'm strongly in favor of reducing tech debt, and moving deliberately > to a > > > 2.0 release. I have a concern with just one processor that is currently > > > marked as deprecated: PostHTTP. (I have not evaluated specifically any > > > other deprecated components; I cannot say if there are or are not > similar > > > issues elsewhere.) I understand the rationale for deprecating this > > > processor in that it eliminates a processor whose functionality is > > > available elsewhere, specifically in the more flexible InvokeHTTP. > > However, > > > in my experience and testing, PostHTTP performs transfers with far > > greater > > > throughput than InvokeHTTP. I would not be in favor of removing > PostHTTP > > > unless/until InvokeHTTP is refactored to increase its throughput > > > capability. > > > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for similar > > > reasons? Or, is there a performance-related configuration for > InvokeHTTP > > I > > > may have missed? > > > > > > Also, in order to help facilitate a smooth transition to 2.0 from a > user > > > perspective, would it be advisable to add some sort of visual indicator > > in > > > the UI for components that are currently annotated as @Deprecated? This > > > might help users proactively modify their flow prior to a release that > > > would otherwise break it. > > > > > > > > > > > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende wrote: > > > > > > > With the merging of MiNiFi and registry into the NiFi repo, we've > > > > actually gone more towards a mono-repo where everything is released > > > > together. Eventually it would still be nice to have a smaller base > > > > distribution containing just the framework and standard NARs, but it > > > > is hard to tackle that until we provide a way for users to easily > > > > obtain the NARs in some other way. > > > > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes > > > > > wrote: > > > > > > > > > > G
Re: [DISCUSS] NiFi 2.0 Release Goals
Thanks for the continued discussion around what can or should be removed. Having a clear migration path from version 1 to version 2 will be very important for supporting current deployments. Following the example of some other projects, one way to consider the upgrade is from the point of view of deprecation warnings. If implemented correctly, an installation of NiFi running the latest minor release of version 1, and showing no deprecation warnings, should be upgradeable to version 2 without any changes. Making this work involves accurately tagging components and methods with deprecation indicators, and providing a clear way to observe deprecation warnings. With the current version containing both deprecated components and deprecated methods in various classes, this would involve some work to get right. A simple approach might be a named logger for deprecation warnings, which would write a separate log file. A more advanced approach might involve a tool to analyze the system and flow configurations. Either way, it seems that some additional work in a minor release version is necessary before considering a major release version. One other point on compatibility: as long as the core component API definition remains backwards compatible, it should be possible to run older components. This provides a transition option for customers using components such as PostHTTP, or any others that are not actively maintained. At some point it may become necessary to break compatibility at that level, but at least for version 2, maintaining component API compatibility is key. Regards, David Handermann On Sun, Aug 1, 2021 at 10:23 AM Mark Bean wrote: > I created a JIRA ticket to investigate and improve the performance of > InvokeHTTP. It includes a flow definition for benchmarking the performance > of both PostHTTP and InvokeHTTP. > > https://issues.apache.org/jira/browse/NIFI-8968 > > Thanks, > Mark > > On Fri, Jul 30, 2021 at 6:41 PM Adam Taft wrote: > > > I'm not seeing the side thread that was going to discuss deprecation of > > PostHTTP. Has that thread started and I just don't see it? > > > > One (significant?) concern with PostHTTP is the smooth integration of > > NiFi-to-NiFi communication that is very transparently enabled with the > > ListenHTTP and PostHTTP processors. There's some special logic in there > for > > handling flowfiles that InvokeHTTP doesn't really (nor should really) > have. > > > > I know of several (large) NiFi installations that rely on the PostHTTP / > > ListenHTTP combination. It has enabled NiFi to NiFi communication for > folks > > reluctant or unable to enable site-to-site type configuration. > > > > Honestly, I don't know why we'd want to "deprecate" any processor, as > > opposed to just moving it to a new location. If these processors can be > > ported and maintained to whatever the 2.0 API looks like, there's > possibly > > little harm keeping them around. > > > > And by the way, what happened to the "marketplace" concept? Is this being > > considered for 2.0 as well? Because relocating the deprecated processors > > to an external nar might be the best solution. Losing PostHTTP entirely I > > think would be a mistake, but I'd conceptually support relocating it. > > > > Thanks, > > > > /Adam > > > > On Tue, Jul 27, 2021 at 2:11 PM Joe Witt wrote: > > > > > Looks like we just need to knock out 5 JIRAs :) [1] > > > > > > I felt like we had a label folks were using at one point but quickly > > > looking revealed nothing exciting. After this confluence page > > > stabilizes a bit we can probably knock out some JIRAs and such. > > > > > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599 > > > > > > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler > > > wrote: > > > > > > > > I find myself wishing I had a list of all the jiras / issues that > have > > > > been put off for a 2.0 release because they required some change or > > > another > > > > :( > > > > > > > > From: Joe Witt > > > > Reply: dev@nifi.apache.org < > dev@nifi.apache.org> > > > > Date: July 27, 2021 at 12:30:35 > > > > To: dev@nifi.apache.org > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > > > A few thoughts: > > > > > > > > 1. I would love to see deprecation notices show up in the UI in > > > > various ways to help motivate users to move off things to more > > > > supportable things. That is not a prerequi
GitHub Workflow Improvements
Team, As part of ongoing efforts to improve automated testing, the following NiFi issues introduced several updates to GitHub workflows: - https://issues.apache.org/jira/browse/NIFI-9037 - https://issues.apache.org/jira/browse/NIFI-9040 - https://issues.apache.org/jira/browse/NIFI-9057 The primary ci-workflow continues to run for all pull requests and pushes to the main branch. In order to avoid unnecessary resource usage, the workflow now includes a concurrency setting that will cancel a running job on a push to the same branch. As a result, multiple updates in quick succession to a single pull request will result in only the last push continuing to build. This also impacts multiple updates to the main branch. When this occurs, GitHub cancels previous jobs for the specified branch with the following message: Canceling since a higher priority waiting request for 'branch' exists In addition to concurrency changes, there is a new system-tests workflow that runs every day at 00:00 UTC: https://github.com/apache/nifi/actions/workflows/system-tests.yml The project README includes an additional badge indicating the system-tests workflow status. The system-tests workflow runs integration tests in the nifi-system-test-suite and nifi-stateless-system-suite modules. These tests are not part of the standard continuous integration workflow, but they provide a very helpful indication of whether NiFi and NiFi stateless are functional at a system level. Reliable automated testing is vital to the continued health of the project, so thanks to everyone who contributes to the testing process! Regards, David Handermann
Re: Flood of JIRAs and presumably PRs to follow for junit-5 migration?
Mike, Thanks for moving things forward on the JUnit 5 migration. I posted a couple comments on the PR for nifi-commons (NIFI-9080) with concerns related to breaking tests and perpetuating ignored unit tests. Summarizing in this thread for general discussion, I am concerned about breaking unit tests as part of the migration process. As long as NiFi supports both JUnit 4 and JUnit 5, the migration should improve the overall testing environment. Breaking existing tests will require additional work to go back and fix, and spending the effort upgrading and reviewing test classes doesn't seem worth it if we continue marking tests as ignored. In some particular cases, it may be worthwhile to remove a test method or test class. For test methods related to performance, migrating to JUnit 5 seems like an ideal opportunity to make tests conditional on environment variables or system properties. I would be glad to help with the migration process, but it would be helpful to avoid having to revisit code multiple times to address these issues. Regards, David Handermann On Wed, Aug 25, 2021 at 2:24 PM Mike Thomsen wrote: > I broke up the tickets because it is A LOT of individual tasks that > can break the overall build and wanted to scope the work appropriately > so that people looking for a chance to contribute could snatch up a > few easy wins. > > I'm still going to take point on making the changes, but the plan > going forward is to submit PRs that bundle a bunch of tickets so that > we don't overwhelm reviewers and the CI/CD pipeline. > > Most of the changes are automated by IntelliJ Ultimate's migration > tool, which from my testing is really good at migrating about 95% of > our JUnit 4 unit and integration tests. > > Thanks, > > Mike > > On Wed, Aug 25, 2021 at 1:05 PM Joe Witt wrote: > > > > Joey > > > > I personally dont care all that much about the number of commits in PR > > - I think that rule is sort of soft already. > > > > I dont think there is any inherent value in having a single module per > > JIRA (and PR or PR commit) on this. These can be done in much coarser > > grained chunks. It will have to be to get review cycles for instance > > (much less having the Github infra to run these builds). > > > > Thanks > > Joe > > > > On Wed, Aug 25, 2021 at 9:44 AM Joey Frazee > > wrote: > > > > > > Maybe this is an exception to the single squashed commit guidance for > the initial pull? > > > > > > I assume the intent is to make incremental progress and not have a PR > with a hundred files affected, but if the different module changes > corresponded to a different commit, GH will make it easy enough to have a > draft and review each commit in isolation. > > > > > > Would that be a reasonable approach? > > > > > > -joey > > > > > > > On Aug 25, 2021, at 9:36 AM, Joe Witt wrote: > > > > > > > > Mike, > > > > > > > > Seeing a pretty stunning flood of JIRAs for 'Refactor nifi-bla to use > > > > JUnit 5' and I'm guessing we'll see the same in terms of PRs. This > is > > > > a really high administrative overhead approach to this. > > > > > > > > Why not break this into one or maybe a few JIRAs/PRs total? > > > > > > > > Thanks >
Re: Preferred approach to the JUnit 5 migration?
Mike, Thanks for raising this question. I have a couple thoughts, and I would be interested in perspectives from others. Having worked on a number of performance and stability issues related to testing, there are varying degrees of test quality throughout the project. The collective result is long-running builds, particularly when it comes to GitHub workflows. Some of these issues require focused effort to correct, which makes it difficult to keep the scope of JUnit 5 migration focused. With that being said, limiting the scope to rearranging imports and expectations does not seem worth the effort. However, using the migration as an opportunity to perform basic cleanup would be very valuable. With that in mind, migration work seems to fit into three categories: 1. Simple replacement of imports: no structural changes needed 2. Refactoring deprecated approaches, such as replacing exception expectations with assertThrows, or replacing platform-specific assumptions with JUnit 5 annotations 3. Removing ignored tests or leveraging JUnit 5 annotations for System property-based enabling For modules that fit into the first category, a larger PR seems like the best way to go, since it should not require more than a cursory review. For modules in the second and third categories, smaller sets of modules are easier to review. With the module-based subtasks that you have already created, using a commit per subtask in a larger PR is a helpful approach. Given that some modules will require more significant refactoring, handling those changes in separate PRs will make it easier to keep the review process moving. Here is what I would like to see in the migration: 1. Elimination of ignored test methods: either complete method removal or changes to use EnabledIfSystemProperty. Using common property names like "nifi.test.performance" would be helpful 2. Elimination of commented-out or bypassed test methods 3. Replacement of platform-specific assumeTrue() references with DisabledOnOs annotations 4. Refactoring of other environment-specific assumptions using utility methods that could be leveraged using EnabledIf or DisabledIf annotations There are other items that would be helpful, but this seems like an opportunity to pay down some technical debt. With these goals in mind, the module scope for pull requests will vary on a case-by-case basis, but that seems like the best use of everyone's time. Thanks for the consideration. Regards, David Handermann On Fri, Aug 27, 2021 at 11:12 AM Mike Thomsen wrote: > Some responders seem to prefer a batched migration while others have > suggested a massive PR that does all of the low-hanging fruit in one > push. I can do either, but would like to know what folks who can do > the reviews would prefer before going much deeper. > > Either way, most of the migration can be automated cleanly with > IntellJ Ultimate's migration tool. > > Thanks, > > Mike >
Re: Pull request for ValidateJSON revisit
Tim, Thanks for highlighting the updated PR, I provided some feedback and would be glad to help move it forward. Regards, David Handermann On Fri, Sep 3, 2021 at 5:18 AM Smith, Tim wrote: > The author submitted a new pull request for ValidateJSON: > https://github.com/apache/nifi/pull/5326 > > Should be ready for a re-review. > > > From: Mark Payne > Sent: Friday, August 20, 2021 9:34:17 AM > To: dev@nifi.apache.org > Subject: Re: Pull request for ValidateJSON revisit > > Tim, > > It looks like I’d done a review but then there were updates and I missed > the fact that the PR had been updated. > > My main concern was with the licensing. It looks like I thought it was MIT > but in fact it was ASL v2 (either I looked at the wrong dependency or the > license was changed). > Looking through its LICENSE and NOTICE file, it doesn’t appear that there > is anything needed in the license. > > At this point, it looks like the PR has been closed, and I cannot re-open > it (Says “The repository that submitted this pull request has been > deleted.”). Based on a quick re-review, I think the PR is okay otherwise. > If you want to open another PR I should be able to quickly review & merge. > > Thanks > -Mark > > > [1] > > > On Aug 20, 2021, at 7:23 AM, Smith, Tim wrote: > > > > The pull request for a ValidateJSON processor, NIFI-7392: > > > > https://github.com/apache/nifi/pull/4232 for > > > > has been marked as stale. From the review comments, this request was > near approval. There was an outstanding question on licensing that still > may exist. I have a similar need for this capability. Could this pull > request be revisited? I would rather not duplicate effort. > > > > > > Tim > >
Re: [ANNOUNCE] New NiFi PMC Member David Handermann
Thank you all for the congratulations! After using NiFi for several years, it is great to be able to contribute back. I look forward to continued involvement on a great project with a great group of collaborators! Regards, David Handermann On Fri, Sep 17, 2021 at 4:25 PM Joe Witt wrote: > So many good things already. Thanks David > > On Fri, Sep 17, 2021 at 12:44 PM Kevin Doran wrote: > > > Congratulations, David! > > > > > On Sep 17, 2021, at 10:09, Joe Gresock wrote: > > > > > > Congrats, David! > > > > > > On Fri, Sep 17, 2021 at 9:57 AM Marton Szasz > wrote: > > > > > >> Congratulations David! > > >> > > >> On Fri, 17 Sept 2021 at 13:07, Chris Sampson > > >> wrote: > > >>> > > >>> Congrats David, very well deserved! > > >>> > > >>> > > >>> On Fri, 17 Sept 2021 at 13:56, Bryan Bende wrote: > > >>> > > >>>> NiFi Community, > > >>>> > > >>>> On behalf of the Apache NiFI PMC, I am very pleased to announce that > > >>>> David Handermann has accepted the PMC's invitation to join the > Apache > > >>>> NiFi PMC. > > >>>> > > >>>> David has made significant improvements in a number of security > > >>>> related areas, he continues to improve the stability of our tests > and > > >>>> CI builds, and regularly helps out the community through the mailing > > >>>> lists, Slack, and PR reviews. > > >>>> > > >>>> Please join us in congratulating and welcoming David to the Apache > > NiFi > > >>>> PMC. > > >>>> > > >>>> Congratulations David! > > >>>> > > >> > > > > >
Re: Single User Authentication Question
Hi Michael, Thanks for raising the question. The Single User Authentication is intended for individual use by definition. NiFi supports integration with a number of services that provide authentication, which avoids some of the complexities surrounding localized management of multiple users. Is there a reason that one of the current integration options is not suitable for your use case? https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization There is an open NiFi Jira issue from several years ago that sounds similar to what you described: https://issues.apache.org/jira/browse/NIFI-1614 Apparently the associated pull request had unresolved feedback and was never merged. If you are interested in this feature, adding a comment about the desired use case would be helpful. This is not a guarantee that the feature will be implemented. The existing multi-tenant integration options cover a wide variety of deployment patterns, but feel free to highlight potential gaps. Regards, David Handermann On Wed, Sep 29, 2021 at 11:57 AM Michael Radov (RIT Alumni) wrote: > Hey, > > Hope all is well. Is there a way to have a future version of nifi include a > way to create multiple single user authentication accounts? IE-->instead of > only 1 single user auth account as that there is now, it would allow the > creation of up to 4-5 accounts and associated management so an admin can > add 4-5 individual accounts. > > How would I get this recommendation added to something that could be > developed into a future NiFi? > > Regards, > Michael Radov >
Re: [VOTE] Release Apache NiFi 1.15.0 (rc1)
-1 (binding) Reproduced the failing system tests described in NIFI-9352. Verified many other features are working as designed, but I agree that this is a blocker for the current release candidate version. Regards, David Handermann On Sat, Oct 30, 2021 at 9:26 AM Mark Payne wrote: > -1 (binding) > > I tried to run the full system test suite and ran into NIFI-9352 [1]. I > think this is a critical enough issue to vote -1 on this release. Good news > is that it's a one-line fix. > > Thanks > -Mark > > [1] https://issues.apache.org/jira/browse/NIFI-9352 > > On 2021/10/29 17:20:44, Joe Witt wrote: > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi 1.15.0. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1184 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/ > > > > A helpful reminder on how the release candidate verification process > works: > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > The Git tag is nifi-1.15.0-RC1 > > The Git commit ID is 73278e2aa0f8673792d069dbf3407faf981adc7c > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=73278e2aa0f8673792d069dbf3407faf981adc7c > > > > Checksums of nifi-1.15.0-source-release.zip: > > SHA256: d7c138219569ba89765d8a16f8c727e7a6ab118744e3689006245412ece59e08 > > SHA512: > ae1a4bd5d86c276535be942bfe6dd4e93ec8ea9d80d5e4abe6ab5c192cf2da6d7a629b55a8aa1dba484a01ecdd64875ff6fc891d9305a2a63db91c794fa08489 > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/joewitt.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 227 issues were closed/resolved for this release: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350382 > > > > Release note highlights can be found here: > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.0 > > > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build from source, and test. > > Then please vote: > > > > [ ] +1 Release this package as nifi-1.15.0 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > > >
Re: [discuss] NiFi 1.15
Chris,
I am running through several verification steps, but I built 1.15.0 RC 3
from sources and then ran "mvn install -pl :dockermaven -P docker" to build
the Docker image. When starting with the following command, I was able to
use the generated username and password to access the running NiFi
container:
docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
If you are still having issues, it would be helpful to provide any logs or
error messages you are seeing.
Regards,
David Handermann
On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran wrote:
> Oh meant to send a link to this too:
>
> > ARG
> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
>
>
> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
>
> > On Nov 4, 2021, at 9:09 PM, Kevin Doran wrote:
> >
> > Hi Joe and Chris,
> >
> > Our Dockerfile that we use to build the Dockerhub image defaults to
> looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we can
> override, so this is easy to workaround incase the release folder does
> change. Agree its nice to keep the tree structure consistent when we can.
> >
> > I’m about to do my verification and will also check the single user with
> the docker image as part of that.
> >
> > Cheers,
> > Kevin
> >
> >> On Nov 4, 2021, at 6:44 PM, Joe Witt wrote:
> >>
> >> Chris,
> >>
> >> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
> >> Should generally not really matter so the docker angle there is
> >> interesting. Not sure why our docker images would have any
> >> relationship to our dist/dev storage. But when I move them into the
> >> release folder I can try to ensure I place them only in 1.15.0 instead
> >> of nifi/1.15.0.
> >>
> >> That directory the prov stuff makes does linger and can be annoying so
> >> thanks for tackling that. Saw the PR. Will take a look soon if
> >> nobody else has.
> >>
> >> Will keep an eye on your findings related to docker builds not working
> >> with username/password things. Hopefully others can chime in there.
> >>
> >> Thanks
> >> Send
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
> >> wrote:
> >>>
> >>> Worryingly, when I do get the Docker Image to build (further changes
> to the
> >>> Dockerfile), the auto-generated username and password from the startup
> logs
> >>> aren't being accepted for login via my browser.
> >>>
> >>> I'll try to spend a little more time looking at this (but await input
> on my
> >>> earlier question/concern also).
> >>>
> >>>
> >>> ---
> >>> *Chris Sampson*
> >>> IT Consultant
> >>> chris.samp...@naimuri.com
> >>>
> >>>
> >>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson
> >>> wrote:
> >>>
> >>>> I've got most of the way through the release check process in order to
> >>>> vote for 1.15.0, but I wanted to check on a change in the distribution
> >>>> release artifacts.
> >>>>
> >>>> For 1.14.0, the Dev artifacts were located at:
> >>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
> >>>> For 1.15.0, the Dev artifacts are located at:
> >>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
> >>>>
> >>>> i.e. there's been a change of directory/path from to
> >>>> nifi-.
> >>>>
> >>>> The reason I raise this is that I can no longer build a Docker Image
> using
> >>>> the dockerhub/DockerBuild.sh script because it cannot find the
> artifacts to
> >>>> download. This may not be a problem if the path change is only for
> the Dev
> >>>> artifacts, but if the same change is going to happen for the released
> >>>> artifacts, then the apache/nifi image (and presumably the
> >>>> apache/nifi-registry, apache/nifi-toolkit and any minifi) convenience
> >>>> images will no longer be possible as part of the Release, which will
> likely
> >>>> be an issue for a number of users that have deployments using these
> images.
> >>>>
> >>>> I spotted this after rebasing m
Re: [VOTE] Release Apache NiFi 1.15.0 (rc3)
+1 (binding) - Built from source on Ubuntu 21.10 and macOS 11.6 with Azul Zulu 11.0.12 and Azul Zulu 1.8.0.282 - Ran NiFi on Azul Zulu 11.0.12 and 1.8.0.282 - Verified Single User Authentication and OIDC Authentication with Keycloak - Verified updates to JWT client storage and cookie handling - Verified basic operation of NiFi Regstriy - NIFI-6617 Verified encrypted repository configuration using new properties with BCFKS and PKCS12 KeyStores - NIFI-7322 Verified SignContentPGP and VerifyContentPGP together with EncryptContentPGP and DecryptContentPGP using various properties - NIFI-8424 Verified absence of HtmlDocumentWriter warnings on startup - NIFI-8943 Built, launched, and restarted Docker container verifying persistence of random sensitive properties key - NIFI-9059 Ran simple data flow using NiFi Stateless on Java 11 - NIFI-9223 Verified ListenSyslog listens on 0.0.0.0 using default properties - NIFI-9251 Verified absence of unused generated password in NiFi Registry - NIFI-9266 Verified Azure Key Vault Secret Sensitive Property Provider configuration for nifi.properties - NIFI-9322 Verified corrected REST API documentation for OIDC and SAML resources - NIFI-9339 Verified JoltTransformJSON custom UI works as expected Thanks for managing the release process Joe! Regards, David Handermann On Thu, Nov 4, 2021 at 6:33 PM Matt Burgess wrote: > +1 Release this package as nifi-1.15.0 > > Ran through the release helper and some tests to verify various Jiras > were included. Thanks again for RM'ing Joe! > > On Wed, Nov 3, 2021 at 3:29 PM Joe Witt wrote: > > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi 1.15.0. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1186 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/ > > > > A helpful reminder on how the release candidate verification process > works: > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > The Git tag is nifi-1.15.0-RC3 > > The Git commit ID is 7fdc07cccdc0e23d4986557a9314e36859704a3b > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=7fdc07cccdc0e23d4986557a9314e36859704a3b > > > > Checksums of nifi-1.15.0-source-release.zip: > > SHA256: 56fe317248941fdbc6216ec973e6ce898d0f682a54e2d063edbabf8542f5288a > > SHA512: > 63f10d9127cf1613c29bf2306be3d7a5b931b31304920706e0df5ea2fe87b58c410efed6ac37afc38d12c65e69f14aec7afb926000fc90cc13f15c738cd15c7f > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/joewitt.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 234 issues were closed/resolved for this release: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350382 > > > > Release note highlights can be found here: > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.0 > > > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build from source, and test. > > Then please vote: > > > > [ ] +1 Release this package as nifi-1.15.0 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... >
Re: [discuss] NiFi 1.15
Thanks for the update Chris! I agree with the suggestion to streamline the Dockerfiles for NiFi builds, there is a lot of duplication between dockermaven and dockerhub, so unifying the modules and parameterizing necessary differences would be a helpful improvement. Regards, David Handermann On Fri, Nov 5, 2021 at 9:10 AM Kevin Doran wrote: > Hi Chris, > > Thanks for the summary of your findings. > > I agree that we should clear up our process for generating Docker images > and that the process should be consistent as possible across NiFi, MiNiFi > Java, Registry, and Toolkit given they are all part of the same repository > and maven project. This will clear up confusion and improve > maintainability. > > I think both of these methods are important and useful: > > 1. building from downloaded convenience binaries for previously released > versions > 2. building from a local assembly output > > That said, I think we could probably consolidate to a single Dockerfile > that takes the binary location as an argument that supports either a URL or > local file path (or a version which could default to the current behavior > of inferring a URL location). This would let us continue to support the > flexibility we have today while maintaining a single Docker file rather > than dockermaven and dockerhub variants. If you and others on the list > agree, I can open up a Jira summarizing what needs to be done. > > Thanks, > Kevin > > > On Nov 5, 2021, at 05:00, Chris Sampson > wrote: > > > > So the good news is that I realised what I was doing wrong yesterday - > I'd > > started a local installation after building the RC, then not shut that > down > > before booting up the Docker instance, so the username/password I was > > trying to use from the Docker logs was wrong against the local > > installation, d'oh! > > > > Having corrected that this morning, I've successfully booted up and > logged > > into a Docker instance built using the RC (with my NIFI-8779 changes so I > > could build from the Dev server instead of the main Apache Archive). > > > > The reason the dockermaven build works is that it uses the locally built > > archive files (i.e. you have to do a full Maven build locally first to > > generate the ZIP files and then create teh Docker image). The > > dockerhub build actually downloads published artifacts from the Apache > > servers - to do this the Dockerfile needs to know the exact path to the > > artifacts it's trying to download, of course. > > > > There was a question recently in one of the Slack channels about whether > > both of these builds (dockerhub and dockermaven) were still valid/needed > - > > I think Joe replied that he wasn't sure (Docker not being an area he > > ventures into much, which is fair enough). It may be worth considering > > whether these two modules are both still needed or can be rationalised > and > > if the latter, which approach should be used - download from the archive > > server or build from local (both have dis/advantages, but the more > > "official" way is arguably to download from the server). > > > > Also maybe worth noting: > > * NiFi Registry only has the "build from local" Dockerfile setup > > * MiniFi (Java) has both local and archive download options, but the > latter > > cannot be used to build an image from the Dev server (the BASE_URL cannot > > be changed via a build arg/env var... this is the issue addressed by > > NIFI-8779 for NiFi) > > * NiFi Toolkit has both local and archive download options, but are > located > > in a single assembly module (instead of split into two like NiFi and > MiniFi) > > > > Assuming the "nifi/" path will be used for the actual release > > artifacts, this shouldn't be a blocker, but it's one to note/remember > when > > the time comes (assuming the "dockerhub" module is what's used to publish > > the "apache/nifi" image to Docker Hub that is). > > > > > > --- > > *Chris Sampson* > > IT Consultant > > chris.samp...@naimuri.com > > > > > > On Fri, 5 Nov 2021 at 03:34, David Handermann < > exceptionfact...@apache.org> > > wrote: > > > >> Chris, > >> > >> I am running through several verification steps, but I built 1.15.0 RC 3 > >> from sources and then ran "mvn install -pl :dockermaven -P docker" to > build > >> the Docker image. When starting with the following command, I was able > to > >> use the generated usernam
Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.3
+1 (binding) - Built nifi-nar-maven-plugin 1.3.3 on Azul Zulu 11.0.10 - Built NiFi using nifi-nar-maven-plugin 1.3.3 - Verified contents of selected extension-manifest.xml files Thanks for managing the release Bryan! Regards, David Handermann On Thu, Dec 2, 2021 at 8:10 AM Pierre Villard wrote: > +1 (binding) > > Le mer. 1 déc. 2021 à 20:30, Otto Fowler a > écrit : > > > +1 (non-binding) > > ran through the testing guide > > > > From: Bryan Bende > > Reply: dev@nifi.apache.org > > Date: November 30, 2021 at 15:56:38 > > To: dev@nifi.apache.org > > Subject: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.3 > > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi NAR Maven Plugin 1.3.3. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1188 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.3/ > > > > A helpful reminder on how the release candidate verification process > works: > > > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > The Git tag is nifi-nar-maven-plugin-1.3.3-RC1 > > The Git commit ID is 99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2 > > > > > https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2 > > > > Checksums of nifi-nar-maven-plugin-1.3.3-source-release.zip: > > SHA256: 36d6579dfdbc4e1450a13457196ff399dbd344624348a62ee5e247cd5962dc77 > > SHA512: > > > > > 0e90100d90e9dd142283aab729957e12cc6abbfdf2f86447f488bcbe5890d55564c19a0efc473307744f813bcdac2636b027ad4e5beebf1ce1ed6ee44deaccbe > > > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/bbende.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 1 issue was closed/resolved for this release: > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12348815 > > > > Release note highlights can be found here: > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.3 > > > > The vote will be open for 72 hours. Please download the release > > candidate and evaluate the necessary items including checking hashes, > > signatures, build from source, and test. Then please vote: > > > > [ ] +1 Release this package as nifi-nar-maven-plugin-1.3.3 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > > >
Re: Nifi 1.14 user authentication using openId connect not working
Hi Ganesh.B, It looks like you are running into an issue in NiFi 1.14.0 that was resolved in 1.15.0 under the following Jira issue: https://issues.apache.org/jira/browse/NIFI-8783 The default authorizers.xml includes a definition for the SingleUserAuthorizer, which can only be used together with the SingleUserLoginIdentityProvider. The workaround for NiFi 1.14.0 is to comment or remove the following section from authorizers.xml: single-user-authorizer org.apache.nifi.authorization.single.user.SingleUserAuthorizer Removing that configuration element from authorizers.xml should allow the OIDC configuration to load as expected in NiFi 1.14.0. Regards, David Handermann On Wed, Dec 8, 2021 at 4:54 AM Ganesh, B (Nokia - IN/Bangalore) < b.gan...@nokia.com> wrote: > Hi , > > > > We are using apache nifi 1.14 . We have 3 nodes in nifi cluster , cluster > is using external zookeeper for state management. > > We are using openId connect for the user authentication . following are > the relevant configuration in nifi.properties file . > > *nifi.security.user.authorizer=managed-authorizer* > > *nifi.security.allow.anonymous.authentication=false* > > *nifi.security.user.login.identity.provider=* > > *.* > > *………..* > > *# OpenId Connect SSO Properties #* > > *nifi.security.user.oidc.discovery.url=https:// SERVER>/access/realms/nifi/.well-known/openid-configuration* > > *nifi.security.user.oidc.connect.timeout=5 secs* > > *nifi.security.user.oidc.read.timeout=5 secs* > > *nifi.security.user.oidc.client.id > <http://nifi.security.user.oidc.client.id>=nifi-client* > > *nifi.security.user.oidc.client.secret=* > > *nifi.security.user.oidc.preferred.jwsalgorithm=RS256* > > > > *But we are observing * > > *org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name > 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': > Unsatisfied dependency expressed through method > 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is > org.springframework.beans.factory.BeanExpressionException: Expression > parsing failed; nested exception is > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied > dependency expressed through method 'setJwtAuthenticationProvider' > parameter 0; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'jwtAuthenticationProvider' defined in class path resource > [nifi-web-security-context.xml]: Cannot resolve reference to bean > 'authorizer' while setting constructor argument; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'authorizer': FactoryBean threw exception on object > creation; nested exception is java.lang.reflect.InvocationTargetException* > > *at > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.resolveMethodArguments(AutowiredAnnotationBeanPostProcessor.java:768)* > > *at > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:720)* > > *at > org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)* > > *at > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)* > > *at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1413)* > > *at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601)* > > *at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)* > > *at > org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)* > > *at > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)* > > *at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)* > > *at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)* > > *at > org.springframework.beans.factory.su
Re: discuss: do a nifi 1.15.1 release to eliminate log4shell concern
Joe, Thanks for starting this discussion. Moving forward with a 1.15.1 patch release sounds like the best path forward. Regards, David Handermann On Mon, Dec 13, 2021 at 7:49 AM Joe Witt wrote: > Team > > We still dont think we are vulnerable but this now highly risky library is > present. We have PRs to eliminate it/main is fixed. I think we should do > a 24 hour 1.15.1 release/vote for it. It will eliminate concerns for > users. We are frankly pretty close to a 1.16 release at this point as > well it seems but can circle back. > > > Any different views on 1.15.1? > > Thanks >
Re: discuss: do a nifi 1.15.1 release to eliminate log4shell concern
PR 5598 for NIFI-9474 is now merged into the main branch, which streamlines version updates to Log4j 2 dependencies. It also excludes log4j-core older than 2.15.0 from build artifacts, so this should provide a good basis for a patch release. https://github.com/apache/nifi/pull/5598 Regards, David Handermann On Mon, Dec 13, 2021 at 10:44 AM Chris Sampson wrote: > I'd agree. The discussions in Slack and separate user mailing list thread > are a reassurance for users (who read them), but a patch for the current > 1.15 branch would seem sensible for people to pick up and assuage any > remaining security concerns they may have around the library. > > That leaves 1.16 a little longer to get more good stuff merged in for the > next feature release. > > > Cheers, > > Chris Sampson > > On Mon, 13 Dec 2021, 14:19 David Handermann, > wrote: > > > Joe, > > > > Thanks for starting this discussion. Moving forward with a 1.15.1 patch > > release sounds like the best path forward. > > > > Regards, > > David Handermann > > > > On Mon, Dec 13, 2021 at 7:49 AM Joe Witt wrote: > > > > > Team > > > > > > We still dont think we are vulnerable but this now highly risky library > > is > > > present. We have PRs to eliminate it/main is fixed. I think we > should > > do > > > a 24 hour 1.15.1 release/vote for it. It will eliminate concerns for > > > users. We are frankly pretty close to a 1.16 release at this point as > > > well it seems but can circle back. > > > > > > > > > Any different views on 1.15.1? > > > > > > Thanks > > > > > >
Re: End of Life for mail-1.4.7.jar file
Hi Sanjeet, NiFi 1.14.0 included changes for the PutEmail processor to use the new Jakarta Mail 2 library as described in NIFI-8630, but I am not aware of a similar Jira issue to change the email notification service that currently references mail-1.4.7.jar. This is easier to do for the bootstrap notification service, but there are a couple other email components that also need to be updated. Addressing each of these upgrades in separate Jira issues is the best way to go. Feel free to write up a Jira issue if you are interested, otherwise this is something that will be considered as time permits. https://issues.apache.org/jira/browse/NIFI-8630 Regards, David Handermann On Tue, Dec 14, 2021 at 7:52 AM sanjeet rath wrote: > Hi > > The latest 1.15 Nifi version contain mail-1.4.7.jar file(inside > \lib\bootstrap folder). > We r using 1.12.1 version of nifi same issue is also there. > > Actually this mail-1.4.7.jar has reached end of life (latest verion was > released on 2013). > > Could someone let me know r we planing to update this jar in future > version. > > Thanks, > Sanjeet >
Re: [ANNOUNCE] New Apache NiFi Committer Margot Tien
Congratulations Margot! On Wed, Dec 15, 2021 at 2:50 PM Nathan Gough wrote: > Congrats Margot, thanks for all your contributions! > > On Wed, Dec 15, 2021 at 3:02 PM Chris Sampson > wrote: > > > Congrat Margot! > > > > --- > > *Chris Sampson* > > IT Consultant > > chris.samp...@naimuri.com > > > > > > On Wed, 15 Dec 2021 at 19:04, Pierre Villard < > pierre.villard...@gmail.com> > > wrote: > > > > > Congrats Margot! > > > > > > Le mer. 15 déc. 2021 à 20:00, Kevin Doran a écrit > : > > > > > > > Congratulations Margot! Well deserved. > > > > > > > > > On Dec 15, 2021, at 13:47, Joe Witt wrote: > > > > > > > > > > Congrats Margot! And thanks > > > > > > > > > > On Wed, Dec 15, 2021 at 11:46 AM Matt Gilman > > > > wrote: > > > > > > > > > >> Apache NiFi community, > > > > >> > > > > >> On behalf of the Apache NiFi PMC, I am very pleased to announce > that > > > > Margot > > > > >> has accepted the PMC's invitation to become a committer on the > > Apache > > > > NiFi > > > > >> project. We greatly appreciate all of Margot's hard work and > > generous > > > > >> contributions to the project. We look forward to continued > > involvement > > > > in > > > > >> the project. > > > > >> > > > > >> Margot has been contributing to NiFi and NiFi Registry for years. > > Her > > > > >> contributions have covered both back-end and front-end > improvements > > in > > > > both > > > > >> projects in addition to release verification and thoughtful PR > > > reviews. > > > > >> > > > > >> Welcome and congratulations! > > > > >> > > > > > > > > > > > > > >
Re: Issue with setting nifi.sensitive.props.key
Phil, The section of the post describing setting the sensitive properties algorithm includes an example toolkit command that can be used to change the sensitive properties algorithm: https://exceptionfactory.com/posts/2021/07/29/deciphering-apache-nifi-component-property-encryption/#setting-the-sensitive-properties-algorithm When upgrading from a previous version of NiFi, you need to start with the previous default value for the algorithm specified in nifi.properties: nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL With that value, you should be able to run the set-sensitive-properties-key command. If you want to change the algorithm to the new default of NIFI_PBKDF2_AES_GCM_256, then you can use the encrypt-config.sh toolkit command described. Regards, David Handermann On Tue, Dec 21, 2021 at 4:43 PM Joe Witt wrote: > Phil > > Not sure if this helps but DavidH wrote this > > https://exceptionfactory.com/posts/2021/07/29/deciphering-apache-nifi-component-property-encryption/#mandatory-sensitive-properties-key > > Thanks > > On Tue, Dec 21, 2021 at 3:38 PM Phil H wrote: > > > > Hi there, > > > > I am in the process of trying to upgrade from 13.2 to 15.1. I did not > have > > a sensitive props key set previously. Based on the upgrade guide, I ran > > > > nifi.sh set-sensitive-properties-key APassword > > > > When I ran nifi, it was complaining about a lack of specified algorithm. > I > > ran up a new installation of 15.1 on another machine which automatically > > specified an algorithm of NIFI_PBKDF2_AES_GCM_256. I copied this value to > > my existing install’s nifi.properties. > > > > When I run nifi now, it halts with a javax.crypto.AEADBadTagException: > mac > > check in GCM failed > > > > If I try the same set-sensitive-properties-key command again, it now > fails > > with the same ‘GCM failed’ exception. If I remove the algorithm line from > > the nifi.properties file, this command works, but then starting nifi > gives > > me an “NullPointerException: Algorithm required” > > > > Not sure what I am missing here! > > > > Help! > > > > Thanks, > > Phil >
Re: [VOTE] Release Apache NiFi 1.15.2
+1 (binding) - Verified hashes and signatures - Built from source on Ubuntu 21.10 Azul Zulu 1.8.0.312 - Ran NiFi on Azul Zulu 1.8.0.312 - Verified Single User Authentication and credentials configuration - NIFI-9483 Verified the absence of Apache Log4j 2 log4j-core in binary distribution - NIFI-9491 Verified the absence of Apache commons-logging in binary distribution - NIFI-9497 Verified Bouncy Castle libraries upgraded to 1.70 - NIFI-9505 Verified Apache Log4j 2 log4-api upgraded to 2.17.0 - NIFI-9507 Verified that SFTP processors stop Keep Alive Threads on failures and run when expected - NIFI-9509 Verified successful SFTP retrieval and renaming using AWS Transfer Family SFTP Server as well as Linux OpenSSH Server Thanks for the quick turnaround on this release Joe! Regards, David Handermann On Tue, Dec 21, 2021 at 8:19 PM Mark Payne wrote: > +1 (binding) > > Verified hashes > Verified signature > Performed full build & verified all unit tests on OS X, OpenJDK 1.8.0 u265 > Successfully ran all 128 System Tests > > Started single standalone instance and verified a few different flows > > Started a two-node secured cluster, did basic verification of permissions, > dataflow running, etc. > > Verified that the convenience binary does not contain the log4j jar in the > lib/ directory or packaged in any NAR. > Verified that the convenience binary does not contain the JndiLookup class > (shaded or otherwise) > > Thanks for putting together another RC Joe! > > Thanks > -Mark > > > On Dec 21, 2021, at 5:51 PM, Joe Witt wrote: > > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi 1.15.2. > > > > This vote is purely bug fix and security focused. This is a > > continuation of our efforts to promptly and thoroughly respond to > > log4shell and related concerns. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1193 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.2/ > > > > A helpful reminder on how the release candidate verification process > works: > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > The Git tag is nifi-1.15.2-RC1 > > The Git commit ID is 1ea460b8556b07057366abb74a5552ace8946e87 > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=1ea460b8556b07057366abb74a5552ace8946e87 > > > > Checksums of nifi-1.15.2-source-release.zip: > > SHA256: 29fcc35c81de80e0fe3f59044e6fbf21bcf523e656aa64914e7546e1d7705e6b > > SHA512: > cabd1f1ad4942a61df0400488d35521202598c217ad8da97dc2d5abe21136604d1f1bb3de9ceb63bb441943de2e29e3515f5cf63607080094e1418d79eb5216b > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/joewitt.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 8 issues were closed/resolved for this release: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351132 > > > > Release note highlights can be found here: > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.2 > > > > Given the nature of the vote and its limited scope > > the vote will be open for 24 hours or until we have sufficient > > votes (instead of the normal 72 hours). > > > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build from source, and test. > > Then please vote: > > > > [ ] +1 Release this package as nifi-1.15.2 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > >
Re: NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments
Hi Sanjeet,
Thank you for providing the stack trace and details of your configuration.
Setting Java System properties in custom code is not a safe or supported
operation in NiFi. As you have observed, setting system properties alters
the behavior of other components, leading to unexpected results.
If you need to support access through a proxy server, the
ProxyConfigurationService interface and StandardProxyConfigurationService
implementation provide a way to specify proxy server properties:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.15.2/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html
Regards,
David Handermann
On Thu, Jan 13, 2022 at 12:39 PM sanjeet rath
wrote:
> Hi,
>
> I encounter one "java.net.NoRouteToHostException: No route to host (Host
> unreachable)" in *Nifi UI login page *.
> The after debugging i realised when i am setting the proxy & port in
> *System.setProperty("proxy address") & System.setProperty("port address")
> *in
> my custom processor. then this issue is appearing .
>
> The other way i also replicated when i am setting at the* jvm lable(in
> bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application this
> exception is coming in nifi ui login page:, After removal of this argument
> it works fine.
>
> Could someone help me to understand what could be the issue.
>
> Nifi version: 1.12.1
> cluster : 3 node (amazon EC2 linux cluster)
>
> *Detail exception from nifi-app.log:*
>
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to "exampledumyserveraddress" due to
> java.net.NoRouteToHostException: No route to host (Host unreachable)
>
> 2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
>
> java.net.NoRouteToHostException: No route to host (Host unreachable)
>
> at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.base/java.net
> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
>
> at
> java.base/java.net
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
>
> at
> java.base/java.net
> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
>
> at
> java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
>
> at java.base/java.net.Socket.connect(Socket.java:609)
>
> at
> okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
>
> at
>
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
>
> at
>
> okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:235)
>
> at
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:177)
>
> at
>
> okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
>
> at
>
> okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
>
> at
> okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
>
> at
> okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
>
> at
>
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
>
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
>
> at
>
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
>
> at
> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
>
> at okhttp3.RealCall.execute(RealCall.java:81)
>
> at
>
> org.apache.nifi
Re: NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments
Hi Sanjeet,
That's correct. Setting the JVM arguments for proxy server access can
create the problems you observed, and it is unlikely to work as expected
when it comes to proxy access. Various NiFi components use different
methods for creating network connections. For this reason, enabling
outgoing proxy server access will be more reliable using the
ProxyConfigurationService. If there specific components that have issues
with proxy server connectivity, it would be worth creating a Jira issue for
those components. If proxy server access is limited to your custom
component, then the best approach is to integrate the
ProxyConfigurationService and determine how that should be wired to your
custom component library.
Regards,
David Handermann
On Thu, Jan 13, 2022 at 1:02 PM sanjeet rath wrote:
> Thanks a lot for the quick response.
>
> So u r suggesting we should not use proxy in jvm argument(nifi
> bootstrap.conf file) lable also as it will impact other component.like in
> my case its impacting nifi-api.
>
> Regards,
> Sanjeet
>
> On Fri, 14 Jan 2022, 12:19 am David Handermann, <
> exceptionfact...@apache.org>
> wrote:
>
> > Hi Sanjeet,
> >
> > Thank you for providing the stack trace and details of your
> configuration.
> > Setting Java System properties in custom code is not a safe or supported
> > operation in NiFi. As you have observed, setting system properties
> alters
> > the behavior of other components, leading to unexpected results.
> >
> > If you need to support access through a proxy server, the
> > ProxyConfigurationService interface and StandardProxyConfigurationService
> > implementation provide a way to specify proxy server properties:
> >
> >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.15.2/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html
> >
> > Regards,
> > David Handermann
> >
> > On Thu, Jan 13, 2022 at 12:39 PM sanjeet rath
> > wrote:
> >
> > > Hi,
> > >
> > > I encounter one "java.net.NoRouteToHostException: No route to host
> > (Host
> > > unreachable)" in *Nifi UI login page *.
> > > The after debugging i realised when i am setting the proxy & port in
> > > *System.setProperty("proxy address") & System.setProperty("port
> address")
> > > *in
> > > my custom processor. then this issue is appearing .
> > >
> > > The other way i also replicated when i am setting at the* jvm
> lable(in
> > > bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application
> this
> > > exception is coming in nifi ui login page:, After removal of this
> > argument
> > > it works fine.
> > >
> > > Could someone help me to understand what could be the issue.
> > >
> > > Nifi version: 1.12.1
> > > cluster : 3 node (amazon EC2 linux cluster)
> > >
> > > *Detail exception from nifi-app.log:*
> > >
> > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> GET
> > > /nifi-api/flow/current-user to "exampledumyserveraddress" due to
> > > java.net.NoRouteToHostException: No route to host (Host unreachable)
> > >
> > > 2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
> > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > >
> > > java.net.NoRouteToHostException: No route to host (Host unreachable)
> > >
> > > at java.base/java.net.PlainSocketImpl.socketConnect(Native
> > Method)
> > >
> > > at
> > > java.base/java.net
> > > .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
> > >
> > > at
> > > java.base/java.net
> > >
> >
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
> > >
> > > at
> > > java.base/java.net
> > > .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
> > >
> > > at
> > > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
> > >
> > > at java.base/java.net.Socket.connect(Socket.java:609)
> > >
> > > at
> > > okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
> > >
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
> > >
> > > at
> > >
> > >
> >
> okhttp3.inter
Re: [VOTE] Release Apache NiFi 1.15.3 (rc1)
+1 (binding) - Verified hashes and signatures - Built from source on Ubuntu 20.04 using Azul Zulu 1.8.0.312 - Ran NiFi on Azul Zulu 1.8.0.312 and 11.0.13 - Verified Single User Authentication and credentials configuration - NIFI-7089 Verified successful build without SFTP test failures - NIFI-7749 Verified ListSFTP operation using authenticated HTTP proxy - NIFI-7835 Verified ListSFTP and FetchSFTP operation using authentication SOCKS proxy - NIFI-9530 Verified absence of google-pubsublite library and successful invocation of ConsumeGCPubSub - NIFI-9534 Verified Log4j 2 API libraries with version 2.17.1 in nifi-elasticsearch-5-nar - NIFI-9566 Verified Logback version 1.2.10 included in NiFi library directory - Started NiFi Registry on Java 11 and created sample bucket for testing - Ran NiFi Encrypt Config Toolkit and verified successful NiFi properties encryption Thanks Joe! Regards, David Handermann On Thu, Jan 13, 2022 at 2:44 PM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache > NiFi 1.15.3. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1194 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.3/ > > A helpful reminder on how the release candidate verification process works: > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.15.3-RC1 > The Git commit ID is 753c311382005acadc16c64952d7b1eaaf2550d5 > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=753c311382005acadc16c64952d7b1eaaf2550d5 > > Checksums of nifi-1.15.3-source-release.zip: > SHA256: 29fcc35c81de80e0fe3f59044e6fbf21bcf523e656aa64914e7546e1d7705e6b > SHA512: > cabd1f1ad4942a61df0400488d35521202598c217ad8da97dc2d5abe21136604d1f1bb3de9ceb63bb441943de2e29e3515f5cf63607080094e1418d79eb5216b > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 11 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351203 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.3 > > The vote will be open > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. > Then please vote: > > [ ] +1 Release this package as nifi-1.15.3 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
Re: Build failure on macOS
Mark, Thanks for highlighting this issue. I have not seen any automated build issues on GitHub, but that appears to be running macOS 11.6.2. I noticed littleshoot 1.1.2 includes a build profile for Netty 4.1. In light of the fact that this works on Linux and other versions of macOS, I wonder if it has something to do with the Netty native libraries. Do you have any additional output in the target directory related to that particular test class? Unfortunately it looks like LittleProxy has not been updated in several years, but it is useful in that test class for verifying connectivity through a proxy server. Regards, David Handermann On Mon, Jan 17, 2022 at 10:07 AM Mark Bean wrote: > Is anyone else having trouble building NiFi (main) on macOS? I recently > upgraded to macOS Monterey 12.1 and since then I have not been able to > build NiFi. I'm not sure if the failure started exactly coincidental with > the macOS upgrade, but it stands out as a significant recent change. > > I get an error on TestHttpClient. > > [ERROR] Error occurred in starting fork, check output in log > [ERROR] Process Exit Code: 134 > [ERROR] Crashed tests: > [ERROR] org.apache.nifi.remote.client.http.TestHttpClient > [ERROR] at > > org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748) > [ERROR] at > > org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:305) > [ERROR] at > > org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:265) > [ERROR] at > > org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314) > [ERROR] at > > org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159) > [ERROR] at > > org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932) > [ERROR] at > > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137) > [ERROR] at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210) > [ERROR] at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156) > [ERROR] at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148) > [ERROR] at > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117) > [ERROR] at > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81) > [ERROR] at > > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56) > [ERROR] at > > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) > [ERROR] at > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305) > [ERROR] at > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192) > [ERROR] at > org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105) > [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957) > [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289) > [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:193) > [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [ERROR] at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > [ERROR] at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [ERROR] at java.lang.reflect.Method.invoke(Method.java:498) > [ERROR] at > > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282) > [ERROR] at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225) > [ERROR] at > > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406) > [ERROR] at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347) > > I think I've traced this back to starting on commit > 12ba579b8f6c506288458fb6cd2191ea26da2cb3. (The commit prior to that does > not have trouble building: b0b1a57b9874cbbbd57122970db96464c7dd1279) > > More specifically, the breaking change introduced > io.netty:netty-all:4.1.72.Final. Previously, it was 4.0.44.Final in the > affected module, nifi-commons/nifi-site-to-site-client.)Replacing the > dependency (transitive in org.littleshoot:littleproxy) with 4.0.44.Final > allows a successful build. > > So, the question is whether this is macOS [12.1] specific, or if there is > something else in my environment. I am using OpenJDK 1.8.0_312, maven > 3.6.3. The build is successful on Linux (Fedora 26). > > Thanks, > Mark >
Re: Nifi toolkit encrypt single properties
Hi Isha, Thanks for contacting the developers list and describing what you are attempting to accomplish. Although there are several Java interfaces and implementation classes that could be leveraged to implement a custom approach to property encryption, that may not be the most reliable method. NiFi does not consider the Sensitive Property Provider interfaces and classes part of the public API, which means they could be subject to change in minor version updates. That being said, the input and output format for encrypted property values will remain supported across versions, so you could use the NiFi classes as a starting point. NiFi stores the serialized encrypted values in a custom format containing the initialization vector, followed by a delimiter, and then the encrypted content. It is possible to use standard AES-GCM support in other languages to read the value if you have the root key available. Recent versions of NiFi introduced support for additional nifi.properties protection schemes using external services such as HashiCorp Vault and Amazon Secrets Manager, as well as other service providers. The effective result is similar in that nifi.properties contains encrypted or placeholder values, but that might provide some different options. For example, the HashiCorp Vault Key Value store implementation uses a placeholder value for the property, instead of the encrypted string. This approach yields the same placeholder value, since HashiCorp Vault stores the actual property. That kind of strategy requires additional infrastructure, so that may not be a good fit for your environment, but it might be worth considering. The most stable integration from the perspective of an external tool is the encrypt-config.sh command and associated arguments. Regards, David Handermann On Tue, Jan 25, 2022 at 2:42 AM Isha Lamboo wrote: > Hi all, > > I hope this question is appropriate for the developers list, if not, I’ll > move it to users. > > I have an Ansible role for NiFi that includes generating the NiFi > properties files from templates and variables set per cluster/host as well > as updating keystore etc. > This works very well, except when it comes to protected properties as set > by the toolkit. The toolkit wants to operate on the actual properties > files, which causes Ansible to then see differences and want to reset. > I tried with an intermediate processing dir, but then I lose the ability > to use Ansible’s –check and –diff options to see if any changes were made. > In the end, I added the encrypted values by hand to the Ansible variables > files. > > As a next step, I’m looking at whether I can use the toolkit’s java > classes in my own wrapper to allow me to pass in a master key, protection > scheme and raw value and get out the encrypted one, so that I can easily > update my encrypted values in the Ansible inventory. > However, my Java skills are somewhat limited, so I would like to ask first > if this is even a good idea. > > Is this a sensible idea or does it conflict with NiFi or encryption design > principles I’m not aware of? > > Kind regards, > > Isha Lamboo > >
Re: [VOTE] Release Apache NiFi Flow Design System 0.3.0 RC2
+1 binding - Ran build using Node 16.13.2 - Verified hashes and signatures - Verified License and Notice files Thanks Scott! Regards, David Handermann On Wed, Jan 26, 2022 at 8:17 AM Shane Ardell wrote: > +1 (non-binding). > > Performed all verification steps listed in the helper guide and ran the > demo app. > > Thanks for RMing, Scott! > > On Wed, Jan 26, 2022 at 9:12 AM Matt Gilman > wrote: > > > +1 (binding) Release this package as nifi-fds-0.3.0 > > > > Ran through release helper. Looks good. Thanks for RMing Scott! > > > > Matt > > > > On Tue, Jan 25, 2022 at 4:41 PM Scott Aslan > wrote: > > > > > Hello, > > > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi > > > Flow Design System nifi-fds-0.3.0. > > > > > > The source zip, including signatures, etc. can be found at: > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-fds/nifi-fds-0.3.0 > > > > > > The Git tag is nifi-0.3.0-RC2 > > > The Git commit ID is a17b6712154f1b7bf36a04cfe728b61640843807 > > > > > > > > > https://gitbox.apache.org/repos/asf?p=nifi-fds.git;a=commit;h=a17b6712154f1b7bf36a04cfe728b61640843807 > > > > > > Checksums of nifi-fds-0.3.0-source-release.zip: > > > SHA1: 92e473c2c0da9f031c47724ade9808f7572dd6bd > > > SHA256: > 200c8f6a2a7fcdfa5a65c73dfef887c0173a9bf10d73ee7ac0160a614e9202ff > > > SHA512: > > > > > > > > > 74b93eb29a5e0bfefd25de288f387308307de63cf159f0c447565615a32b041fe5af20a326528a5dc8e99bf3adc8bbdc157cfa10090d51b30f958bb6253201b9 > > > > > > Release artifacts are signed with the following key: > > > https://people.apache.org/keys/committer/scottyaslan.asc > > > > > > KEYS file available here: > > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > > > 17 issues were closed/resolved for this release: > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12345959 > > > > > > Release note highlights can be found here: > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiFlowDesignSystem0.3.0 > > > > > > The vote will be open for 72 hours. > > > Please download the release candidate and evaluate the necessary items > > > including checking hashes, signatures, build from source, and test. > Then > > > please vote: > > > > > > [ ] +1 Release this package as nifi-fds-0.3.0 > > > [ ] +0 no opinion > > > [ ] -1 Do not release this package because... > > > > > >
Re: [discuss] pulling together a NiFi 1.16
Mark, To elaborate on Joe's reply, changing the trust store configuration alters the security profile of NiFi by allowing clients with trusted certificates to access the system. Changing the key store and trust store should always occur in conjunction with changing the authorization configuration. The Single User Authorizer includes a safety check to prevent configuration in conjunction with Login Identity Providers other than the Single User Login Identity Provider. In the case of certificate authentication, however, the Login Identity Provider does not apply, since certificate authentication happens prior to application-level access. Although it might be possible to add further safety checking in the Single User Authorizer to also check the username against a particular value, this could introduce additional coupling between authentication and authorization. As this is primarily a configuration problem, and changing the trust store has a significant impact on the security profile of the system, this particular scenario does not seem like a significant concern. Regards, David Handermann On Wed, Mar 9, 2022 at 9:55 AM Joe Witt wrote: > Mark > > The single user authorizer and default setup install is just to avoid > having wide open systems by default. So if you want to make changes to > security settings and do it right you dont' use that mode. Happy to have > improvements within that scope of intent but does not sound like anything > we'd wait for. When it lands it lands. > > Thanks > > On Wed, Mar 9, 2022 at 8:49 AM Mark Bean wrote: > > > Joe, > > > > I just discovered an issue yesterday that might need attention first. I > > haven't investigated fully yet nor created a ticket because I don't yet > > fully understand it. However, it appears as though the > > single-user-authorizer may not be behaving as intended. When I updated > > nifi.properties to swap the self-signed, auto-generated keystore and > > truststore with "real" ones, single-user became _every_ user. My > suspicion > > is that any user whose browser presents a cert that was signed by a CA in > > the truststore is allowed in - without even prompting for > > username/password. > > > > It may be considered a configuration error to allow this to happen. > Still, > > this seems like extremely dangerous behavior. > > > > -Mark > > > > > > On Wed, Mar 9, 2022 at 10:42 AM Joe Witt wrote: > > > > > Team > > > > > > We appear to be at a good point to start pulling together the release > > > candidate for 1.16. > > > > > > https://issues.apache.org/jira/projects/NIFI/versions/12350741 > > > > > > I'm basically waiting for > > https://issues.apache.org/jira/browse/NIFI-9761 > > > to land then will start pulling together the release. > > > > > > Thanks > > > > > > On Mon, Feb 14, 2022 at 11:18 AM Joe Witt wrote: > > > > > > > Eduardo > > > > > > > > Getting reviewers on the UI/rest/front-end are among the toughest as > > > > there just aren't as many of those folks. > > > > > > > > The reply from Pierre was probably most telling. It looks fine but > > > > many of us would pause to merge without knowing precisely what the > > > > implications are. What happens on a taxed system with many > > > > CSs...I''ll comment on the PR. > > > > > > > > Thanks > > > > Joe > > > > > > > > On Mon, Feb 14, 2022 at 11:13 AM Eduardo Fontes > > > > wrote: > > > > > > > > > > Hi All, > > > > > > > > > > Is it possible to include > > > > https://issues.apache.org/jira/browse/NIFI-8927 > > > > > in release 1.16? > > > > > I've been asking for a review > > https://github.com/apache/nifi/pull/5247 > > > > > since AUG/2021 and I don't understand why nobody did it. It's a > > simple > > > > and > > > > > useful UI feature. > > > > > > > > > > Peace out. > > > > > Eduardo Fontes > > > > > > > > > >
Re: nifi.apache.org website
Hi Steven, Thanks for your interest! This would also be a great opportunity to streamline the HTML generation approach to use something like Hugo or similar static-site generation system. Regards, David Handermann On Wed, Mar 9, 2022 at 12:44 PM Joe Witt wrote: > Steven > > To say this is super welcome is an understatement. We definitely need the > facelift you mention and we need to get off the old website mechanism and > onto the newer one the ASF wants projects using. This would be awesome! > > Thanks! > > > On Wed, Mar 9, 2022 at 11:40 AM Steven Matison > wrote: > > > Hey Dev Team, > > > > Is there any way I could assist with modernizing the > nifi.apache.org > > website? > > > > Its 2022 and I think a facelift is in order... I have a ton of > experience > > with web-dev and super passionate about NiFi. This would be something I > > would love to be a part of. > > > > Thanks, > > > > Steven > > >
Re: [VOTE] Release Apache NiFi 1.16.0 (rc2)
-1 (binding) In light of a problem just reported with AccessToken.isExpired() determination in the StandardOauth2TokenProvider service, this issue should be corrected: https://issues.apache.org/jira/browse/NIFI-9801 Regards, David Handermann On Wed, Mar 16, 2022 at 12:05 PM Mark Payne wrote: > +1 (binding) > > - Performed full build with contib-chck > - Veified the hashes > - Ran all system tests successfully > - Started up and loaded a large flow with thousdands of processors. > Ensured that all loaded successfully. > - Ran some smoke screen tests > - Ran several tests with disconnecting nodes, updated flow, ensuring that > node could join back and properly inherited all changes > > Thanks > -Mark > > > > On Mar 15, 2022, at 4:17 PM, Joe Witt wrote: > > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > NiFi > > 1.16.0. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1197 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.0/ > > > > A helpful reminder on how the release candidate verification process > works: > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > The Git tag is nifi-1.16.0-RC2 > > The Git commit ID is a680b3f093fc93197036fb43c08e75f6a3f21a18 > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=a680b3f093fc93197036fb43c08e75f6a3f21a18 > > > > Checksums of nifi-1.16.0-source-release.zip: > > SHA256: 02c03615df5be60958717471e40ca97d1b47457ef7355e056b385e3384cb54f2 > > SHA512: > > > 43e9baa695b9617999d88f4d5878775d37bc293747c0cd6bcb1fc77318fca07bde5c5c536cc273937593c0194dd25426a617137861832b1b3749307eb2b9fd70 > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/joewitt.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 387 issues were closed/resolved for this release: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350741 > > > > Release note highlights can be found here: > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.0 > > > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build from source, and test. Then > > please vote: > > > > [ ] +1 Release this package as nifi-1.16.0 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > >
[ANNOUNCE] New Apache NiFi Committer Paul Grey
Apache NiFi community, On behalf of the Apache NiFi PMC, I am very pleased to announce that Paul Grey has accepted the PMC's invitation to become a committer on the Apache NiFi project. Paul has contributed a number of pull requests and code reviews over the past year, improving project security and test stability in a number of areas. We appreciate Paul's work and look forward to continued contributions! Welcome Paul, and congratulations!
Java 17 and GitHub Workflow Updates
Development Team, Thanks to reviews from Mike Thomsen and Paul Grey, support for building the NiFi project on Java 17 is now in the main branch. The GitHub continuous integration workflow now includes a fourth environment: Ubuntu Zulu JDK 17 EN Although some dependencies may cause runtime issues with Java 17, this new build target will help highlight and correct particular issues. For the purpose of compatibility, restricted reflective access is one of the most important differences in Java 17. Java 11 flags these issues as illegal reflective access warnings, but Java 17 prevents these operations. New issues with particular components should be flagged and addressed using the standard Jira tracking process. This is also a good time to mention ongoing work related to JUnit 5. Big thanks to Mike Thomsen for working through numerous modules and multiple rounds of feedback to update a majority of components to JUnit 5! All new test code should use JUnit 5 org.junit.jupiter components. Some work still remains to convert various modules, which can be tracked through sub-tasks under the following Jira issue: https://issues.apache.org/jira/browse/NIFI-9084 Finally, shout out to Paul Grey for his efforts to improve the reliability of the NiFi system tests on GitHub. The resource-constrained GitHub runners can expose test stability problems that are difficult to track down, but thanks to Paul's work, the stability of the system tests on GitHub has improved. Regards, David Handermann
Re: [CANCEL][VOTE] Release Apache NiFi 1.16.0 (rc2)
Mike, Thanks for reviewing and merging the OAuth2-related pull requests, as well as the Java 17 build PR! I agree we should be ready to move forward now. Regards, David Handermann On Fri, Mar 18, 2022 at 7:35 AM Mike Thomsen wrote: > I just finished testing David's patch against one of my client's > OAuth2-enabled REST APIs (OAuth2 provider is Keycloak). Merged his > patch based on the good results I saw. I think we're good to go now > unless there are other blockers. > > On Wed, Mar 16, 2022 at 2:51 PM Joe Witt wrote: > > > > RC2 is cancelled. > > > > On Wed, Mar 16, 2022 at 11:46 AM David Handermann < > > exceptionfact...@apache.org> wrote: > > > > > -1 (binding) > > > > > > In light of a problem just reported with AccessToken.isExpired() > > > determination in the StandardOauth2TokenProvider service, this issue > should > > > be corrected: > > > > > > https://issues.apache.org/jira/browse/NIFI-9801 > > > > > > Regards, > > > David Handermann > > > > > > On Wed, Mar 16, 2022 at 12:05 PM Mark Payne > wrote: > > > > > > > +1 (binding) > > > > > > > > - Performed full build with contib-chck > > > > - Veified the hashes > > > > - Ran all system tests successfully > > > > - Started up and loaded a large flow with thousdands of processors. > > > > Ensured that all loaded successfully. > > > > - Ran some smoke screen tests > > > > - Ran several tests with disconnecting nodes, updated flow, ensuring > that > > > > node could join back and properly inherited all changes > > > > > > > > Thanks > > > > -Mark > > > > > > > > > > > > > On Mar 15, 2022, at 4:17 PM, Joe Witt wrote: > > > > > > > > > > Hello, > > > > > > > > > > I am pleased to be calling this vote for the source release of > Apache > > > > NiFi > > > > > 1.16.0. > > > > > > > > > > The source zip, including signatures, digests, etc. can be found > at: > > > > > > https://repository.apache.org/content/repositories/orgapachenifi-1197 > > > > > > > > > > The source being voted upon and the convenience binaries can be > found > > > at: > > > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.0/ > > > > > > > > > > A helpful reminder on how the release candidate verification > process > > > > works: > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > > > > > > > The Git tag is nifi-1.16.0-RC2 > > > > > The Git commit ID is a680b3f093fc93197036fb43c08e75f6a3f21a18 > > > > > > > > > > > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=a680b3f093fc93197036fb43c08e75f6a3f21a18 > > > > > > > > > > Checksums of nifi-1.16.0-source-release.zip: > > > > > SHA256: > > > 02c03615df5be60958717471e40ca97d1b47457ef7355e056b385e3384cb54f2 > > > > > SHA512: > > > > > > > > > > > > > 43e9baa695b9617999d88f4d5878775d37bc293747c0cd6bcb1fc77318fca07bde5c5c536cc273937593c0194dd25426a617137861832b1b3749307eb2b9fd70 > > > > > > > > > > Release artifacts are signed with the following key: > > > > > https://people.apache.org/keys/committer/joewitt.asc > > > > > > > > > > KEYS file available here: > > > > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > > > > > > > 387 issues were closed/resolved for this release: > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350741 > > > > > > > > > > Release note highlights can be found here: > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.0 > > > > > > > > > > The vote will be open for 72 hours. > > > > > Please download the release candidate and evaluate the necessary > items > > > > > including checking hashes, signatures, build from source, and test. > > > Then > > > > > please vote: > > > > > > > > > > [ ] +1 Release this package as nifi-1.16.0 > > > > > [ ] +0 no opinion > > > > > [ ] -1 Do not release this package because... > > > > > > > > > > > >
Re: Workflow approval for first time contributor
Hi Jon, I just approved the workflow run. Regards, David Handermann On Mon, Mar 21, 2022 at 9:47 AM Jonathan Conti-Vock < jonathan.contiv...@gmail.com> wrote: > Hi Devs, > > As a first-time code contributor, my pull request [1] requires a maintainer > to approve running workflows. Could someone with that authority please > approve this? > > [1] https://github.com/apache/nifi/pull/5857 > > Cheers, > > Jon >
Re: [VOTE] Release Apache NiFi 1.16.0 (rc3)
+1 (binding) - Verified signatures and hashes - Ran build using Maven 3.8.4 - Ran build on Ubuntu 20.04 with Azul Zulu JDK 1.8.0_322 - Ran build on Ubuntu 20.04 with Azul Zulu JDK 17.0.2 - Ran system tests - Ran NiFi Stateless on Azul Zulu JDK 17.0.2 - Verified generation of random sensitive properties key - Verified execution of simple logging flow - Ran NiFi Registry on Azul Zulu JDK 17.0.2 - Created Buckets - Imported and Exported Flows - Integrated and synchronized sample flow - NIFI-9630 Verified NiFi Registry REST API documentation - NIFI-9375 Verified NiFi Registry avoids illegal reflective access on startup - Ran NiFi on Azul Zulu JDK 17.0.2 and 1.8.0_322 - Verified encryption of nifi.properties using encrypt-config - NIFI-830 Verified URL path filename strategy for InvokeHTTP - NIFI-1825 Verified replay of 0 byte FlowFiles - NIFI-5684 Verified space indicator on property values containing spaces - NIFI-6699 Verified ListSFTP handling of symbolic links for directories and files - NIFI-7749 Verified SFTP connection with HTTP authenticated proxy - NIFI-7835 Verified SFTP connection with SOCKS authenticated proxy - NIFI-8806 Verified ListenTCP with TLS using PutTCP sender - NIFI-9227 Verified Run Once works with CRON Driven scheduling - NIFI-9291 Verified HTTP request logging - NIFI-9368 Verified descending bulletin ordering on Processors - NIFI-9412 Verified generation of sensitive properties key for MiNiFi - NIFI-9543 Verified bring to front capability for Labels - NIFI-9585 Verified H2 2.1.210 successful migration from H2 1.4 - NIFI-9688 Verified shutdown messages in logs - NIFI-9724 Verified set-sensitive-properties-algorithm command - NIFI-9735 Verified absence of Jetty Duplicate Mapping warning - NIFI-9711 Verified flow.json.gz handled using set-sensitive-properties-key - NIFI-9734 Verified adjustments to exception cause bulletin messages - NIFI-9747 Verified PID on shutdown logs - NIFI-9782 Verified absence of H2 library from nifi-druid-nar - NIFI-9785 Verified login-identity-providers.xml update processing - NIFI-9807 Verified Standard OAuth2 Access Token Provider with Refresh Window property Thanks Joe! Regards, David Handermann On Tue, Mar 22, 2022 at 5:50 PM Nathan Gough wrote: > +1 (non-binding), looks good to me. > > Ran through the release helper, tested a few flows, tested a secure cluster > with external ZK, and tested OIDC auth. > > On Tue, Mar 22, 2022 at 5:59 PM Matt Burgess wrote: > > > +1 Release this package as nifi-1.16.0 > > > > Ran through release helper, ran a couple flows through NiFi including > > one for PutDatabaseRecord (to validate the changes), verified MiNiFi > > started and autogenerated the sensitive propert(ies), verified H2 > > upgrade on NiFi Registry. > > > > Thanks for RM'ing Joe! > > > > On Mon, Mar 21, 2022 at 5:42 PM Joe Witt wrote: > > > > > > Hello, > > > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi > > > 1.16.0. > > > > > > The source zip, including signatures, digests, etc. can be found at: > > > https://repository.apache.org/content/repositories/orgapachenifi-1198 > > > > > > The source being voted upon and the convenience binaries can be found > at: > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.0/ > > > > > > A helpful reminder on how the release candidate verification process > > works: > > > > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > > > The Git tag is nifi-1.16.0-RC3 > > > The Git commit ID is b019a9191f1c83bc7f547dc02c1b679b8936acee > > > > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b019a9191f1c83bc7f547dc02c1b679b8936acee > > > > > > Checksums of nifi-1.16.0-source-release.zip: > > > SHA256: > 2f16cb94df404d1bcc9c32835ba98da8940005a5d7ea5504c155ee42021a221e > > > SHA512: > > > > > > cbd95f15cec5ffe506fef204526267b18b77d7266f6dc3e1bbc3c7600aac12e119977f7d8cf93dbbbc86fbb0739ba88aaa11a5381d29a463ec9a0c9a18f4e9e6 > > > > > > Release artifacts are signed with the following key: > > > https://people.apache.org/keys/committer/joewitt.asc > > > > > > KEYS file available here: > > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > > > 401 issues were closed/resolved for this release: > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350741 > > > > > > Release note highlights can be found here: > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.0 > > > > > > The vote will be open for 72 hours. > > > Please download the release candidate and evaluate the necessary items > > > including checking hashes, signatures, build from source, and test. > Then > > > please vote: > > > > > > [ ] +1 Release this package as nifi-1.16.0 > > > [ ] +0 no opinion > > > [ ] -1 Do not release this package because... > > >
Re: CVE-2022-22965: Spring RCE
Hi Tristan, Although NiFi 1.15.3 and earlier include Spring Framework libraries identified with CVE-2022-22965, initial research suggests that NiFi is not impacted. NiFi and NiFi Registry use Jetty, whereas the vulnerability requires running applications on Apache Tomcat. The vulnerability also involves data binding connected with spring-webmvc and spring-webflux, but NiFi uses JAX-RS with Jersey for REST request handling. If further research uncovers additional attack vectors, that could change the analysis. NiFi has already upgraded the current main branch to use Spring Framework 5.3.18 and Spring Boot 2.6.6, which will be incorporated in upcoming releases. Please see the following NiFi Jira issue for additional details regarding the upgrade and background on the vulnerability: https://issues.apache.org/jira/browse/NIFI-9852 Regards, David Handermann On Thu, Mar 31, 2022 at 7:38 PM Tristan Steele wrote: > Good Day, > > I've been reading through some of the information that is now available > about the recently reported remote code execution vulnerability in the > Spring framework and it appears that a vulnerable version of this library > is part of the 1.15.3 release? > > Is it known yet if this library is used in a way that makes it vulnerable > to exploitation? Will there likely be a new release that updates this > dependency to one that is not affected? > > Thanks in advance for any assistance on this one, > Tristan >
Re: [discuss] nifi 1.16.1
Joe, Thanks for wrapping up the 1.16.0 release and preparing 1.16.1, there are a handful of bug fixes and dependency updates that will be helpful to include as a part of 1.16.1. I am in the process of reviewing NIFI-7253 upgrading Jackson, it looks like it should be ready, pending successful automated builds. The Avro upgrade for NIFI-7234 covers more ground in terms of upgrading versions, and appears to involve more changes. We definitely should move in that direction, and I think there have been some previous efforts to upgrade Avro dependencies. I can take an initial look at the PR soon, but it would be helpful for others to give it more attention. There are a couple other dependency upgrades I plan to put forward soon, so perhaps waiting until Wednesday would give a bit more time to get some of those completed. Regards, David Handermann On Mon, Apr 4, 2022 at 3:45 PM Mike Thomsen wrote: > I have a few PRs for standardizing and updating our dependencies like > Jackson and Avro. Might be good to get those included in 1.16.1 > > On Mon, Apr 4, 2022 at 3:02 PM Joe Witt wrote: > > > > Team, > > > > Sorry for the delays in wrapping up the 1.16 release from earlier last > > week. We had great vote turnout and it was definitely a large release in > > terms of features/etc.. > > > > Watching evolution on the 1.17 line since it is clear we can benefit > from a > > 1.16.1 release so I've started preparing that and will closely monitor > > candidates for inclusion. > > > > The branch is pushed and is called 'support/nifi-1.16'. I've already > > pulled in everything on 1.17 that looks like it can work in a pure 3rd > > digit release. Some of this makes sense due to bugs found and or natural > > concerns of scanners for things like springshell/etc.. In any case it is > > always good to tighten up the release lines if time permits. > > > > I'll probably start the release process tomorrow or Wednesday. Will > watch > > for anything that comes in. > > > > Thanks >
Re: Missing maven dependencies when building nifi
Phil, Recent versions of NiFi require Java 8 Update 251 or newer in order to support modern signing algorithms. Java 8 Update 171 and newer include the unrestricted Java Cryptography Extension Policy. Upgrading to the latest Java 8 version should resolve the illegal key size issue shown in the build. Regards, David Handermann On Wed, Apr 6, 2022 at 6:16 AM Phil H wrote: > Australia. Never had any other issues with OpenSSL stuff, etc > > On Wed, 6 Apr 2022 at 21:11, Otto Fowler wrote: > > > What country are you in? Are you under export controls? > > > > From: Phil H > > Reply: dev@nifi.apache.org > > Date: April 6, 2022 at 07:08:45 > > To: dev@nifi.apache.org > > Subject: Re: Missing maven dependencies when building nifi > > > > So, I restarted the whole process inside a CentOS VM. The only way I > could > > get the build to complete was by using -DskipTests in the maven command. > > > > Trying to run the built nifi fails after maybe 15 seconds (see below): > > > > [phil@localhost nifi-1.17.0-SNAPSHOT]$ bin/nifi.sh run > > > > Java home: /usr/java/jdk1.8.0_131/ > > NiFi home: /home/phil/nifi-1.17.0-SNAPSHOT > > > > Bootstrap Config File: > /home/phil/nifi-1.17.0-SNAPSHOT/conf/bootstrap.conf > > > > 2022-04-06 21:02:19,292 INFO [main] org.apache.nifi.bootstrap.Command > > Generating Self-Signed Certificate: Expires on 2022-06-05 > > 2022-04-06 21:02:20,426 ERROR [main] org.apache.nifi.bootstrap.Command > > Self-Signed Certificate Generation Failed > > java.io.IOException: exception encrypting data - > > java.security.InvalidKeyException: Illegal key size > > at > > > > > org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.wrapKey(Unknown > > > > Source) > > at > > > > > org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.doStore(Unknown > > > > Source) > > at > > > > > org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.engineStore(Unknown > > > > Source) > > at > > > > > org.bouncycastle.jcajce.provider.keystore.util.AdaptingKeyStoreSpi.engineStore(Unknown > > > > Source) > > at java.security.KeyStore.store(KeyStore.java:1377) > > at > > > > > org.apache.nifi.security.util.KeyStoreUtils.createKeyStoreAndGetX509Certificate(KeyStoreUtils.java:540) > > > > at > > > > > org.apache.nifi.security.util.KeyStoreUtils.createTlsConfigAndNewKeystoreTruststore(KeyStoreUtils.java:228) > > > > at > > > > > org.apache.nifi.bootstrap.util.SecureNiFiConfigUtil.configureSecureNiFiProperties(SecureNiFiConfigUtil.java:123) > > > > at org.apache.nifi.bootstrap.RunNiFi.start(RunNiFi.java:1249) > > at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:294) > > 2022-04-06 21:02:20,428 INFO [main] org.apache.nifi.bootstrap.Command > > Starting Apache NiFi... > > 2022-04-06 21:02:20,428 INFO [main] org.apache.nifi.bootstrap.Command > > Working Directory: /home/phil/nifi-1.17.0-SNAPSHOT > > 2022-04-06 21:02:20,428 INFO [main] org.apache.nifi.bootstrap.Command > > Command: /usr/java/jdk1.8.0_131/bin/java -classpath > > > > > /home/phil/nifi-1.17.0-SNAPSHOT/./conf:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/javax.servlet-api-3.1.0.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/jetty-schemas-5.2.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/logback-classic-1.2.11.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/logback-core-1.2.11.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/jcl-over-slf4j-1.7.36.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/jul-to-slf4j-1.7.36.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/log4j-over-slf4j-1.7.36.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/slf4j-api-1.7.36.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-api-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-framework-api-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-server-api-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-runtime-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-nar-utils-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-properties-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-property-utils-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-stateless-bootstrap-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-stateless-api-1.17.0-SNAPSHOT.jar > > > > -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m > > -Dcurator-log-only-first-connection-issue-as-error-level=true > > -Djavax.security.auth.useSubjectCredsOnly=true > > -Djava.security.egd=file:/dev/urandom > -Dzookeeper.ad
Re: Missing maven dependencies when building nifi
Updating the minimum Java version in the build configuration would be helpful, I will look into submitting a pull request to include this check. Regards, David Handermann On Wed, Apr 6, 2022 at 7:23 AM Chakravarty, G wrote: > Just wondering if Nifi runs through any pre-install checks like minimum > java version, OS etc. like many other software. Will be nice if it does > basic checks to prevent common install failures. > ____ > From: David Handermann > Sent: Wednesday, April 6, 2022 8:11 AM > To: dev@nifi.apache.org > Subject: Re: Missing maven dependencies when building nifi > > Phil, > > Recent versions of NiFi require Java 8 Update 251 or newer in order to > support modern signing algorithms. Java 8 Update 171 and newer include the > unrestricted Java Cryptography Extension Policy. Upgrading to the latest > Java 8 version should resolve the illegal key size issue shown in the > build. > > Regards, > David Handermann > > On Wed, Apr 6, 2022 at 6:16 AM Phil H wrote: > > > Australia. Never had any other issues with OpenSSL stuff, etc > > > > On Wed, 6 Apr 2022 at 21:11, Otto Fowler > wrote: > > > > > What country are you in? Are you under export controls? > > > > > > From: Phil H > > > Reply: dev@nifi.apache.org > > > Date: April 6, 2022 at 07:08:45 > > > To: dev@nifi.apache.org > > > Subject: Re: Missing maven dependencies when building nifi > > > > > > So, I restarted the whole process inside a CentOS VM. The only way I > > could > > > get the build to complete was by using -DskipTests in the maven > command. > > > > > > Trying to run the built nifi fails after maybe 15 seconds (see below): > > > > > > [phil@localhost nifi-1.17.0-SNAPSHOT]$ bin/nifi.sh run > > > > > > Java home: /usr/java/jdk1.8.0_131/ > > > NiFi home: /home/phil/nifi-1.17.0-SNAPSHOT > > > > > > Bootstrap Config File: > > /home/phil/nifi-1.17.0-SNAPSHOT/conf/bootstrap.conf > > > > > > 2022-04-06 21:02:19,292 INFO [main] org.apache.nifi.bootstrap.Command > > > Generating Self-Signed Certificate: Expires on 2022-06-05 > > > 2022-04-06 21:02:20,426 ERROR [main] org.apache.nifi.bootstrap.Command > > > Self-Signed Certificate Generation Failed > > > java.io.IOException: exception encrypting data - > > > java.security.InvalidKeyException: Illegal key size > > > at > > > > > > > > > org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.wrapKey(Unknown > > > > > > Source) > > > at > > > > > > > > > org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.doStore(Unknown > > > > > > Source) > > > at > > > > > > > > > org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.engineStore(Unknown > > > > > > Source) > > > at > > > > > > > > > org.bouncycastle.jcajce.provider.keystore.util.AdaptingKeyStoreSpi.engineStore(Unknown > > > > > > Source) > > > at java.security.KeyStore.store(KeyStore.java:1377) > > > at > > > > > > > > > org.apache.nifi.security.util.KeyStoreUtils.createKeyStoreAndGetX509Certificate(KeyStoreUtils.java:540) > > > > > > at > > > > > > > > > org.apache.nifi.security.util.KeyStoreUtils.createTlsConfigAndNewKeystoreTruststore(KeyStoreUtils.java:228) > > > > > > at > > > > > > > > > org.apache.nifi.bootstrap.util.SecureNiFiConfigUtil.configureSecureNiFiProperties(SecureNiFiConfigUtil.java:123) > > > > > > at org.apache.nifi.bootstrap.RunNiFi.start(RunNiFi.java:1249) > > > at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:294) > > > 2022-04-06 21:02:20,428 INFO [main] org.apache.nifi.bootstrap.Command > > > Starting Apache NiFi... > > > 2022-04-06 21:02:20,428 INFO [main] org.apache.nifi.bootstrap.Command > > > Working Directory: /home/phil/nifi-1.17.0-SNAPSHOT > > > 2022-04-06 21:02:20,428 INFO [main] org.apache.nifi.bootstrap.Command > > > Command: /usr/java/jdk1.8.0_131/bin/java -classpath > > > > > > > > > /home/phil/nifi-1.17.0-SNAPSHOT/./conf:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/javax.servlet-api-3.1.0.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/jetty-schemas-5.2.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/logback-classic-1.2.11.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/logback-core-1.2.11.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/jcl-over-slf4j-1.7.36.ja
Re: Missing maven dependencies when building nifi
Phil, You're welcome. I have also updated the NiFi Contributor Guide to mention minimum supported versions: https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide The latest version of Java 8 is currently update 322. The automated builds on GitHub run on Azul Zulu and Eclipse Temurin. Any standard OpenJDK-derived version should work. Regards, David Handermann On Wed, Apr 6, 2022 at 2:31 PM Phil H wrote: > Much appreciated. If I am upgrading the JDK on this VM (which I will only > use for NiFi contributions) should I be moving to a more modern version, > and if so, what is recommended? > > On Wed, 6 Apr 2022 at 22:32, David Handermann > > wrote: > > > Updating the minimum Java version in the build configuration would be > > helpful, I will look into submitting a pull request to include this > check. > > > > Regards, > > David Handermann > > > > On Wed, Apr 6, 2022 at 7:23 AM Chakravarty, G > wrote: > > > > > Just wondering if Nifi runs through any pre-install checks like minimum > > > java version, OS etc. like many other software. Will be nice if it does > > > basic checks to prevent common install failures. > > > > > > From: David Handermann > > > Sent: Wednesday, April 6, 2022 8:11 AM > > > To: dev@nifi.apache.org > > > Subject: Re: Missing maven dependencies when building nifi > > > > > > Phil, > > > > > > Recent versions of NiFi require Java 8 Update 251 or newer in order to > > > support modern signing algorithms. Java 8 Update 171 and newer include > > the > > > unrestricted Java Cryptography Extension Policy. Upgrading to the > latest > > > Java 8 version should resolve the illegal key size issue shown in the > > > build. > > > > > > Regards, > > > David Handermann > > > > > > On Wed, Apr 6, 2022 at 6:16 AM Phil H wrote: > > > > > > > Australia. Never had any other issues with OpenSSL stuff, etc > > > > > > > > On Wed, 6 Apr 2022 at 21:11, Otto Fowler > > > wrote: > > > > > > > > > What country are you in? Are you under export controls? > > > > > > > > > > From: Phil H > > > > > Reply: dev@nifi.apache.org < > > dev@nifi.apache.org> > > > > > Date: April 6, 2022 at 07:08:45 > > > > > To: dev@nifi.apache.org > > > > > > Subject: Re: Missing maven dependencies when building nifi > > > > > > > > > > So, I restarted the whole process inside a CentOS VM. The only way > I > > > > could > > > > > get the build to complete was by using -DskipTests in the maven > > > command. > > > > > > > > > > Trying to run the built nifi fails after maybe 15 seconds (see > > below): > > > > > > > > > > [phil@localhost nifi-1.17.0-SNAPSHOT]$ bin/nifi.sh run > > > > > > > > > > Java home: /usr/java/jdk1.8.0_131/ > > > > > NiFi home: /home/phil/nifi-1.17.0-SNAPSHOT > > > > > > > > > > Bootstrap Config File: > > > > /home/phil/nifi-1.17.0-SNAPSHOT/conf/bootstrap.conf > > > > > > > > > > 2022-04-06 21:02:19,292 INFO [main] > org.apache.nifi.bootstrap.Command > > > > > Generating Self-Signed Certificate: Expires on 2022-06-05 > > > > > 2022-04-06 21:02:20,426 ERROR [main] > > org.apache.nifi.bootstrap.Command > > > > > Self-Signed Certificate Generation Failed > > > > > java.io.IOException: exception encrypting data - > > > > > java.security.InvalidKeyException: Illegal key size > > > > > at > > > > > > > > > > > > > > > > > > > > org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.wrapKey(Unknown > > > > > > > > > > Source) > > > > > at > > > > > > > > > > > > > > > > > > > > org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.doStore(Unknown > > > > > > > > > > Source) > > > > > at > > > > > > > > > > > > > > > > > > > > org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.engineStore(Unknown > > > > > > > > > > Source) > > > > > at > > > > > > > > >
Re: New build problem, Java 17
Phil, Can you provide the version of Maven you are using? Running Maven with the -X argument for debug will generate a lot of logging information, but it might point to the POM resolution problem. Regards, David Handermann On Wed, Apr 6, 2022 at 8:13 PM Phil H wrote: > I reverted back to AZUL JDK 11, and then in turn to Oracle JDK 11, and all > had the same issue. > > So I’m assuming it’s either a OSX thing, or some sort of weird maven issue? > > > On 7 Apr 2022, at 10:02 am, Mike Thomsen wrote: > > > >> However it’s not a network issue because… > > > > Could be something funky with how Java 17's TLS APIs are working on your > setup. > > > > Java 8 and Java 11 are your best bets for now anyway. Java 17 support > > is still in the early stages and isn't a preferred JDK/JRE. > > > > On Wed, Apr 6, 2022 at 7:00 PM Phil H wrote: > >> > >> So I upgraded to the Azul JDK 17. New repo, following the same > instructions as before. > >> > >> phil@Phils-MacBook-Pro nifi % mvn -T C2.0 clean install > -Pinclude-grpc > >> [INFO] Scanning for projects... > >> Downloading from central: > https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom > >> Downloading from apache-repo: > https://repository.apache.org/content/repositories/releases/org/apache/apache/25/apache-25.pom > >> Downloading from Shibboleth: > https://build.shibboleth.net/nexus/content/repositories/releases/org/apache/apache/25/apache-25.pom > >> [ERROR] [ERROR] Some problems were encountered while processing the > POMs: > >> [FATAL] Non-resolvable parent POM for > org.apache.nifi:nifi:1.17.0-SNAPSHOT: Could not transfer artifact > org.apache:apache:pom:25 from/to central (https://repo1.maven.org/maven2): > transfer failed for > https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom and > 'parent.relativePath' points at no local POM @ line 14, column 13 > >> @ > >> [ERROR] The build could not read 1 project -> [Help 1] > >> [ERROR] > >> [ERROR] The project org.apache.nifi:nifi:1.17.0-SNAPSHOT > (/Users/phil/nifi/pom.xml) has 1 error > >> [ERROR] Non-resolvable parent POM for > org.apache.nifi:nifi:1.17.0-SNAPSHOT: Could not transfer artifact > org.apache:apache:pom:25 from/to central (https://repo1.maven.org/maven2): > transfer failed for > https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom and > 'parent.relativePath' points at no local POM @ line 14, column 13: Network > is unreachable -> [Help 2] > >> [ERROR] > >> [ERROR] To see the full stack trace of the errors, re-run Maven > with the -e switch. > >> [ERROR] Re-run Maven using the -X switch to enable full debug > logging. > >> [ERROR] > >> [ERROR] For more information about the errors and possible > solutions, please read the following articles: > >> [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException > >> [ERROR] [Help 2] > http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException > >> > >> However it’s not a network issue because… > >> > >> phil@Phils-MacBook-Pro nifi % curl > https://repo1.maven.org/maven2/org/apache/apache/25/apache-25.pom > >> > >> > >>
Re: New build problem, Java 17
Phil, Thanks for the additional details, Maven 3.8.1 is recent, so it should work. The stack trace include the following specific problem: - java.net.SocketException: Network is unreachable (connect failed) This indicates a low-level network connectivity problem. Although you provided the curl command results, is it possible that your network connection requires a proxy server? Running the following OpenSSL command will attempt a socket connection with TLS to the Maven repository server, to check connectivity: openssl s_client -host repo1.maven.org -port 443 Regards, David Handermann On Wed, Apr 6, 2022 at 8:31 PM Phil H wrote: > Sure thing David, > > phil@Phils-MacBook-Pro nifi % mvn -X -T C2.0 clean install > -Pinclude-grpc > Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d) > Maven home: /usr/local/Cellar/maven/3.8.1/libexec > Java version: 11.0.12, vendor: Oracle Corporation, runtime: > /Library/Java/JavaVirtualMachines/jdk-11.0.12.jdk/Contents/Home > Default locale: en_AU, platform encoding: UTF-8 > OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac" > [DEBUG] Created new class realm maven.api > [DEBUG] Importing foreign packages into class realm maven.api > [DEBUG] Imported: javax.annotation.* < plexus.core > [DEBUG] Imported: javax.annotation.security.* < plexus.core > [DEBUG] Imported: javax.enterprise.inject.* < plexus.core > [DEBUG] Imported: javax.enterprise.util.* < plexus.core > [DEBUG] Imported: javax.inject.* < plexus.core > [DEBUG] Imported: org.apache.maven.* < plexus.core > [DEBUG] Imported: org.apache.maven.artifact < plexus.core > [DEBUG] Imported: org.apache.maven.classrealm < plexus.core > [DEBUG] Imported: org.apache.maven.cli < plexus.core > [DEBUG] Imported: org.apache.maven.configuration < plexus.core > [DEBUG] Imported: org.apache.maven.exception < plexus.core > [DEBUG] Imported: org.apache.maven.execution < plexus.core > [DEBUG] Imported: org.apache.maven.execution.scope < plexus.core > [DEBUG] Imported: org.apache.maven.lifecycle < plexus.core > [DEBUG] Imported: org.apache.maven.model < plexus.core > [DEBUG] Imported: org.apache.maven.monitor < plexus.core > [DEBUG] Imported: org.apache.maven.plugin < plexus.core > [DEBUG] Imported: org.apache.maven.profiles < plexus.core > [DEBUG] Imported: org.apache.maven.project < plexus.core > [DEBUG] Imported: org.apache.maven.reporting < plexus.core > [DEBUG] Imported: org.apache.maven.repository < plexus.core > [DEBUG] Imported: org.apache.maven.rtinfo < plexus.core > [DEBUG] Imported: org.apache.maven.settings < plexus.core > [DEBUG] Imported: org.apache.maven.toolchain < plexus.core > [DEBUG] Imported: org.apache.maven.usability < plexus.core > [DEBUG] Imported: org.apache.maven.wagon.* < plexus.core > [DEBUG] Imported: org.apache.maven.wagon.authentication < plexus.core > [DEBUG] Imported: org.apache.maven.wagon.authorization < plexus.core > [DEBUG] Imported: org.apache.maven.wagon.events < plexus.core > [DEBUG] Imported: org.apache.maven.wagon.observers < plexus.core > [DEBUG] Imported: org.apache.maven.wagon.proxy < plexus.core > [DEBUG] Imported: org.apache.maven.wagon.repository < plexus.core > [DEBUG] Imported: org.apache.maven.wagon.resource < plexus.core > [DEBUG] Imported: org.codehaus.classworlds < plexus.core > [DEBUG] Imported: org.codehaus.plexus.* < plexus.core > [DEBUG] Imported: org.codehaus.plexus.classworlds < plexus.core > [DEBUG] Imported: org.codehaus.plexus.component < plexus.core > [DEBUG] Imported: org.codehaus.plexus.configuration < plexus.core > [DEBUG] Imported: org.codehaus.plexus.container < plexus.core > [DEBUG] Imported: org.codehaus.plexus.context < plexus.core > [DEBUG] Imported: org.codehaus.plexus.lifecycle < plexus.core > [DEBUG] Imported: org.codehaus.plexus.logging < plexus.core > [DEBUG] Imported: org.codehaus.plexus.personality < plexus.core > [DEBUG] Imported: org.codehaus.plexus.util.xml.Xpp3Dom < plexus.core > [DEBUG] Imported: org.codehaus.plexus.util.xml.pull.XmlPullParser < > plexus.core > [DEBUG] Imported: > org.codehaus.plexus.util.xml.pull.XmlPullParserException < plexus.core > [DEBUG] Imported: org.codehaus.plexus.util.xml.pull.XmlSerializer < > plexus.core > [DEBUG] Imported: org.eclipse.aether.* < plexus.core > [DEBUG] Imported: org.eclipse.aether.artifact < plexus.core > [DEBUG] Imported: org.eclipse.aether.collection < plexus.core > [DEBUG] Imported: org.eclipse.aether.deployment < plexus.core > [DEBUG] Imported: org.eclipse.aether.graph < plexus.
Re: Need some feedback on how upgrading Avro might cause problems
Mike,
Thanks for raising this issue for discussion, and thanks Isha and Edward
for the comments. It is helpful to get an understanding of the potential
impact.
In general, it seems like moving forward with the upgrade is the best idea,
given the issues with older versions. For integration with other products
outside of NiFi, using the latest version of Avro should provide better
compatibility.
That being said, it is important to consider the impact on existing
deployments. At minimum, this change is something that should be described
in the NiFi Migration Guidance associated with the released version. As far
as supporting different versions, it is possible to run older versions of
the various components, although this is more complicated given the scope
of the upgrade. This seems like a case where providing more detailed
migration steps would be helpful, to include issues related to cached
schemas or inferred schemas used as the basis for customization.
Regards,
David Handermann
On Thu, Apr 7, 2022 at 6:08 AM Edward Armes wrote:
> I've had a quick look in JIRA and it looks like this might have happened as
> a side effect of AVRO-1544.
>
> I think it is worth upgrading especially given that it looks like few of
> the changes refer to updating against newer bundled dependencies some of
> which seem to have CVE's against them.
>
> Depending on peoples feelings would it be wroth creating 2 versions one
> using Avro 1.8 and one using Avro 1.11.0 and then removing the 1.8 version
> in a later release?
>
> On an additional note in cases where people are writing schemas manually I
> suspect they are probably going to be validating against the later versions
> of Avro using the projects tooling and that may create issues further down
> the line.
>
> Edward
>
> On Thu, Apr 7, 2022 at 11:52 AM Isha Lamboo <
> isha.lam...@virtualsciences.nl>
> wrote:
>
> > Hi Mike,
> >
> > The "Infer schema" functionality in NiFi currently generates schemas with
> > the order that will be invalid under Avro 1.9+. I noticed because I've
> been
> > using that to copy-paste schemas that were "almost right" so I could
> > manually fix them.
> >
> > I guess that inferred schemas should be fine if the inferring logic is
> > also upgraded by the dependency, but for any cached schemas and my own
> > manually saved schemas this will be somewhat painful.
> > My use case for manually saving inferred schemas is mostly database
> > migration where some inferred "choice" fields are not supported for the
> > target database.
> >
> > Hope this helps,
> >
> > Isha
> >
> > -Oorspronkelijk bericht-
> > Van: Mike Thomsen
> > Verzonden: donderdag 7 april 2022 12:11
> > Aan: dev@nifi.apache.org
> > Onderwerp: Need some feedback on how upgrading Avro might cause problems
> >
> > Thread is here for full details:
> >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F5900%23pullrequestreview-922490039&data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C8c89ae3e621c4a255c3308da187eea09%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637849231546931433%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ztcxOWXBkpZFqEh%2Bu%2BG0du5BLUPyZ3WaMxqpeqn%2FexI%3D&reserved=0
> >
> > It looks like Avro 1.8's schema parser may have been more lenient (or
> > buggy) in enforcing the specification with respect to the ordering of a
> > union for a nullable type. 1.9.X and higher are definitely more
> opinionated
> > and throw exceptions on schemas that used to work on 1.8.X. For example,
> > this used to be valid:
> >
> > {
> > "name": "message",
> > "type": [ "null", "string" ],
> > "default": "Hello, world"
> > }
> >
> > Now Avro **correctly** throws an exception per the specification.
> > Under 1.8 it did not, and as you can see here, I had to change numerous
> > test schemas in order to make them work under 1.9 to 1.11.0:
> >
> >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F5900%2Ffiles%23r835954170&data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C8c89ae3e621c4a255c3308da187eea09%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637849231546931433%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=iBGW6sYZUdxvAADYIB5L2t94RBZH3A5%2BPJMhxuGv8q8%3D&reserved=0
> >
> > As I said to Matt, I think we're in a "damned if you do, damned if you
> > don't" position here.
> >
> > Thoughts?
> >
>
Re: Anyone else seeing this on macOS with the RC1 build?
Mike, I observed the same behavior when attempting to build on Java 17. The problem appears specific to Java 17, and does not appear on Java 8 or 11. The root cause appears to be that the `user.timezone` System property is not set by default on Java 17, causing System.setProperty() to fail with the NPE. I was able to work around the problem by adding "-DargLine=user.timezone=SomeTimeZone" to the Maven build command. This does not impact GitHub Java 17 builds since those builds set user.timezone. Since this is a build problem specific to Java 17, it does not seem to be a blocking issue, in my opinion. Regards, David Handermann On Mon, Apr 25, 2022 at 4:10 PM Mike Thomsen wrote: > This laptop can be a little wonky at times because of the corporate > lockdowns, so I wanted to see if any other committers/PMC members have > seen this in nifi-record-path: > > [INFO] Running org.apache.nifi.record.path.util.TestFieldValueWalker > > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.083 s - in org.apache.nifi.record.path.util.TestFieldValueWalker > > [INFO] Running > org.apache.nifi.record.path.util.TestFieldValueLogicalPathBuilder > > [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 0.007 s - in > org.apache.nifi.record.path.util.TestFieldValueLogicalPathBuilder > > [INFO] Running org.apache.nifi.record.path.TestRecordPath > > [ERROR] Tests run: 76, Failures: 0, Errors: 1, Skipped: 0, Time > elapsed: 0.336 s <<< FAILURE! - in > org.apache.nifi.record.path.TestRecordPath > > [ERROR] org.apache.nifi.record.path.TestRecordPath Time elapsed: > 0.336 s <<< ERROR! > > java.lang.NullPointerException > > at > java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011) > > at > java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) > > at java.base/java.util.Properties.put(Properties.java:1301) > > at java.base/java.util.Properties.setProperty(Properties.java:229) > > at java.base/java.lang.System.setProperty(System.java:999) > > at > org.apache.nifi.record.path.TestRecordPath.setSystemTimezone(TestRecordPath.java:82) > > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) > > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:568) >
Re: [VOTE] Release Apache NiFi 1.16.1
+1 (binding) - Verified signatures and hashes - Ran build using Maven 3.8.5 - Ran build on Ubuntu 20.04 with Azul Zulu JDK 1.8.0-332 - Ran build on Ubuntu 20.04 with Azul Zulu JDK 11.0.15 - Ran build on Ubuntu 20.04 with Azul Zulu JDK 17.0.3 - Ran system tests on Azul Zulu JDK 17.0.3 - Ran stateless system tests on Azul Zulu JDK 17.0.3 - Ran NiFi Stateless on Azul Zulu JDK 17.0.3 - Verified execution of simple logging flow - Ran NiFi Registry on Azul Zulu JDK 17.0.3 - Created Buckets - Imported and Exported Flows - Ran NiFi on Azul Zulu JDK 17.0.3 and 1.8.0-332 - Verified encryption of nifi.properties using encrypt-config - NIFI-8124 Verified InvokeHTTP works as designed with ACCEPT_ALL Cookie Strategy - NIFI-9816 Verified presence of lang attribute on several HTML pages - NIFI-9820 Verified default Worker Count matches available processors in PutKudu - NIFI-9824 Verified absence of warning log when running a processor - NIFI-9862 Verified JsonTreeReader Starting Field Strategy with Root Node and Nested values - NIFI-9870 Verified Jetty upgraded to 9.4.46 - NIFI-9871 Verified correct formatting of Bulletins with causes and logs with stack traces - NIFI-9882 Verified HTML documentation generated without errors - NIFI-9883 Verified separation of property-protection modules in properties directory - NIFI-9901 Verified several flows using EvaluateXPath and EvaluateXQuery processors - NIFI-9919 Verified SFTP RSA Public Key authentication works with Azure Blob Storage SFTP - NIFI-9923 Verified component logs contain throwable causes in Bulletin messages - NIFI-9954 Verified Spring Framework upgraded to 5.3.19 Thanks Joe! Regards, David Handermann On Mon, Apr 25, 2022 at 1:29 PM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.16.1. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1200 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.1/ > > A helpful reminder on how the release candidate verification process works: > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.16.1-RC1 > The Git commit ID is 81166797e552b9d14b482807632f2f04321b2018 > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=81166797e552b9d14b482807632f2f04321b2018 > > Checksums of nifi-1.16.1-source-release.zip: > SHA256: 2c39b45ba8eec42d601d5c9facad623d14660dd209c011b4a13b2b559b9f3e32 > SHA512: > > cd670ab558937cac709dea0b4be3351559f57c9e9aedf54d5153706eee386a021262ef199e2bf9485763cf955931edfd6a24ca1c5a0748a77e3eeb91c490cbda > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 83 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351504 > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.1 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. Then > please vote: > > [ ] +1 Release this package as nifi-1.16.1 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
Re: Anyone else seeing this on macOS with the RC1 build?
Mike, Following up on this thread, I created the following Jira issue and pull request to address the problem for future builds on Java 17: https://issues.apache.org/jira/browse/NIFI-9968 https://github.com/apache/nifi/pull/6000 Regards, David Handermann On Mon, Apr 25, 2022 at 4:32 PM David Handermann < exceptionfact...@apache.org> wrote: > Mike, > > I observed the same behavior when attempting to build on Java 17. The > problem appears specific to Java 17, and does not appear on Java 8 or 11. > The root cause appears to be that the `user.timezone` System property is > not set by default on Java 17, causing System.setProperty() to fail with > the NPE. I was able to work around the problem by adding > "-DargLine=user.timezone=SomeTimeZone" to the Maven build command. This > does not impact GitHub Java 17 builds since those builds set user.timezone. > > Since this is a build problem specific to Java 17, it does not seem to be > a blocking issue, in my opinion. > > Regards, > David Handermann > > On Mon, Apr 25, 2022 at 4:10 PM Mike Thomsen > wrote: > >> This laptop can be a little wonky at times because of the corporate >> lockdowns, so I wanted to see if any other committers/PMC members have >> seen this in nifi-record-path: >> >> [INFO] Running org.apache.nifi.record.path.util.TestFieldValueWalker >> >> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: >> 0.083 s - in org.apache.nifi.record.path.util.TestFieldValueWalker >> >> [INFO] Running >> org.apache.nifi.record.path.util.TestFieldValueLogicalPathBuilder >> >> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: >> 0.007 s - in >> org.apache.nifi.record.path.util.TestFieldValueLogicalPathBuilder >> >> [INFO] Running org.apache.nifi.record.path.TestRecordPath >> >> [ERROR] Tests run: 76, Failures: 0, Errors: 1, Skipped: 0, Time >> elapsed: 0.336 s <<< FAILURE! - in >> org.apache.nifi.record.path.TestRecordPath >> >> [ERROR] org.apache.nifi.record.path.TestRecordPath Time elapsed: >> 0.336 s <<< ERROR! >> >> java.lang.NullPointerException >> >> at >> java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011) >> >> at >> java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) >> >> at java.base/java.util.Properties.put(Properties.java:1301) >> >> at java.base/java.util.Properties.setProperty(Properties.java:229) >> >> at java.base/java.lang.System.setProperty(System.java:999) >> >> at >> org.apache.nifi.record.path.TestRecordPath.setSystemTimezone(TestRecordPath.java:82) >> >> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >> at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) >> >> at >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.base/java.lang.reflect.Method.invoke(Method.java:568) >> >
CVE-2022-29265: Apache NiFi: Improper Restriction of XML External Entity References in Multiple Components
Severity: moderate Description: Multiple components in Apache NiFi 0.0.1 to 1.16.0 do not restrict XML External Entity references in the default configuration. The Standard Content Viewer service attempts to resolve XML External Entity references when viewing formatted XML files. The following Processors attempt to resolve XML External Entity references when configured with default property values: - EvaluateXPath - EvaluateXQuery - ValidateXml Apache NiFi flow configurations that include these Processors are vulnerable to malicious XML documents that contain Document Type Declarations with XML External Entity references. The resolution disables Document Type Declarations in the default configuration for these Processors, and disallows XML External Entity resolution in standard services. This issue is being tracked as NIFI-9901 Mitigation: Disabling the Validate DTD Processor Property in EvaluateXPath and EvaluateXQuery mitigates the vulnerability for those Processors. No mitigation is available for the ValidateXml Processor or the Standard Content Viewer. Credit: David Handermann at exceptionfactory.com reported this issue. References: https://nifi.apache.org/security.html#CVE-2022-29265
Re: [discuss] nifi 1.16.2
Joe, Thanks for initiating this discussion and moving toward 1.16.2. The following two issues should be good candidates to include in 1.16.2, among other potentials: https://issues.apache.org/jira/browse/NIFI-9988 - Corrected handling of property decryption in authorizers.xml and login-identity-providers.xml https://issues.apache.org/jira/browse/NIFI-9980 - Corrected Google dependencies for GCP components Regards, David Handermann On Mon, May 9, 2022 at 11:39 AM Joe Witt wrote: > Team, > > Just saw this one land https://issues.apache.org/jira/browse/NIFI-9993. > And while the underlying issue has been around a while this also seems > likely something that could create a lot of disruption. I'm planning to > spin up a 1.16.2 to address it since I'm not sure how long until 1.17.0. > > I'll look for other things that have landed which fit and spin up an RC in > the next day or two but will keep an eye here for anything else flagged > that might make sense. > > Thanks >
Re: Assistance in Upgrading Nifi to 1.15.3
Hi Nathan, Please subscribe to the developers mailing list in order to receive further replies. See the NiFi Mailing Lists page for details on subscribing: https://nifi.apache.org/mailing_lists.html Regarding the decryption problem described, either the Sensitive Properties Key (nifi.sensitive.props.key) or the Sensitive Properties Algorithm (nifi.sensitive.props.algorithm) in nifi.properties does not match the value used when NiFi previously saved flow.xml.gz. It is difficult to provide additional details given the custom configuration described, but as a starting point, all NiFi nodes must have the same value for both the Sensitive Properties Key and Sensitive Properties Algorithm in the copy of nifi.properties. When nifi.sensitive.props.key is empty and NiFi is configured to run in standalone mode, NiFi will generate a random key on startup. If you were previously running NiFi 1.12.1 without a value for nifi.sensitive.props.key, then you need to use either encrypt-config.sh or nifi.sh set-sensitive-properties-key to set a new value. Unless you used the encrypt-config.sh command, you need to retain the previous algorithm value set in nifi.sensitive.props.algorithm. The following post provides some additional background on sensitive properties, as well as the changes introduced in NiFi 1.14.0: https://exceptionfactory.com/posts/2021/07/29/deciphering-apache-nifi-component-property-encryption/ Regards, David Handermann On Wed, May 11, 2022 at 5:07 PM wrote: > Classification: UNCLASSIFIED > == > > Good afternoon, > > Our team is in the midst of a Nifi upgrade from 1.12.1 to 1.15.3 and we > are encountering issues installing our 1.12.1 flow.xml.gz to 1.15.3 via > cloud formation template. > > Our process of starting up Nifi is a bit different than what I've seen > online, in that we have the Nifi 1.15.3 tar in S3 along with it's > corresponding /conf files and /lib folder holding our custom nar file. The > cloud formation script pulls a install script from S3 that pulls and > installs Nifi in an EC2 instance. Once installed, we sync the S3 folders > holding our /conf and /lib files into the Nifi EC2's conf and lib folder, > set ownership to local user nifi, and start Nifi. > > For the upgrade from 1.12.1 to 1.15.3, we have to account for the > encryption update that was introduced in 1.14.0. What we did to mitigate > the upgrade was decrypt the sensitive values in the 1.12.1 flow.xml.gz file > w/ the old algorithm and key, and encrypt the same sensitive values using > the new algorithm and key generated inside the nifi.properties file from a > flow-less Nifi 1.15.3. Once we've set the sensitive values to the new > algorithm, we place the newly modified flow.xml.gz into a new S3 bucket, > copy over the conf files and lib nar into a new 1.15.3 bucket, and stand up > a new cloud formation template pointing to the new location of the conf and > lib files. > > While this worked on my local machine and in the dedicated developer test > environment, we are having issues trying to apply the same logic in our > staging environment. For some reason, we've noticed that when we pull the > conf folder containing our new 1.15.3 flow, some or all encrypted sensitive > values in flow.xml.gz were different than what we've set it up prior to > sending it up to S3, causing a [AES/GES/NoPadding] error right after it > starts the flow controller in the nifi-app logs. > > We also had the approach of using an existing nifi.properties file w/ key, > placing it into S3 and running the same encryption steps to set the > flow.xml.gz to the current algorithm and while this worked locally, it also > failed with the same decryption error. > > Looking online, the encrypt-config.sh approach did not work despite > defining the correct parameters. We plan on utilizing templates tomorrow to > see if that approach will work. > > Any assistance is much appreciated, > - Nathan Velasquez > == > Classification: UNCLASSIFIED > >
Re: [VOTE] Release Apache NiFi 1.16.2 (RC3)
+1 (binding) - Verified signatures and hashes - Ran build using Maven 3.8.5 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-332 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 11.0.15 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 17.0.3 - Ran system tests on Azul Zulu JDK 17.0.3 - Ran stateless system tests on Azul Zulu JDK 17.0.3 - Ran NiFi Stateless on Azul Zulu JDK 17.0.3 - Verified execution of simple logging flow - Ran NiFi Registry on Azul Zulu JDK 17.0.3 - Created Buckets - Imported and Exported Flows - NIFI-10014 Ran all database verification tests - Ran NiFi on Azul Zulu JDK 17.0.3 and 1.8.0-332 - NIFI-9977 Verified presence of optional scope parameter requested for OAuth2 Access Token Provider - NIFI-9984 Verified Standard OAuth2 Access Token Provider accepts HTTP 201 success codes - NIFI-9988/NIFI-10018 Verified encryption and decryption of values in login-identity-providers.xml and authorizers.xml - NIFI-9990 Verified FetchFTP with ProFTPD routes to not.found as expected - NIFI-10010 Verified presence of certificate attributes on ListenTCP FlowFiles - NIFI-10024 Verified Netty 4.1.77 with ListenTCP - NIFI-10031/NIFI-10026/NIFI-10020 - Verified email bootstrap notifications - NIFI-10036 Verified PutElasticsearchRecord with Elasticsearch 7.17.3 Thanks Joe! Regards, David Handermann On Sun, May 22, 2022 at 10:44 PM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.16.2. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1203 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.2/ > > A helpful reminder on how the release candidate verification process works: > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.16.2-RC3 > The Git commit ID is 06f04958272dafc30ce357c4c4edcaf470050b52 > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=06f04958272dafc30ce357c4c4edcaf470050b52 > > Checksums of nifi-1.16.2-source-release.zip: > SHA256: 1fecf7d9f6001cc8e58d4a46ece08e141de705bcd227338ba79e9cb574267415 > SHA512: > > 1f4fd4e5e9f24949830a75949b302a67b8826049406ab8296c4b8c99a5a0aa1d211f84f98699b3af6fb41efa305f35a3f85b21d7958dc09c027cc1ed836c169f > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 34 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351721 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.2 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. Then > please vote: > > [ ] +1 Release this package as nifi-1.16.2 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
Re: Migrating from 1.15 to 1.16.2 - ParameterContext inheritance problem
Steve, Thanks for flagging this issue. I commented on the Jira issue after reproducing the same problem, and I am evaluating a solution. Regards, David Handermann On Thu, Jun 9, 2022 at 12:49 PM Steven Youtsey wrote: > When migrating to 1.16.2, we noticed that we lost some parameter context > inheritance relationships. I filed a JIRA ticket for this [1], but I was > hoping that someone else in this community may have some insight on this. > > Thanks, > Steve > > [1] https://issues.apache.org/jira/browse/NIFI-10096 > > > Steven Youtsey > SS&C Technologies, Inc. > >
Re: Upgrading Cassandra driver and bringing in breaking changes with it
Mike, Thanks for digging into this issue! I agree that simplifying the implementation makes going forward. The newer driver version involves changes in the Controller Service method signatures either way, so streamlining the connection handling approach would be helpful. Based on feedback, perhaps summarizing the planned implementation in the Jira issue would be helpful. Regards, David Handermann On Fri, Jun 10, 2022 at 3:06 PM Mike Thomsen wrote: > https://issues.apache.org/jira/browse/NIFI-9770 > > Based on David's description saying it will require some deep changes > to how various properties are configured, I think this ticket would be > a good time to simplify the Cassandra bundle in general to just use a > controller service to configure the connection. (Current approach is > controller service or manual configuration on a per-processor basis). > > Thoughts? >
Re: [VOTE] Release Apache NiFi 1.16.3 (RC1)
+1 (binding) - Verified signatures and hashes - Ran build using Maven 3.8.5 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-332 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 11.0.15 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 17.0.3 - Ran system tests on Azul Zulu JDK 17.0.3 - Ran stateless system tests on Azul Zulu JDK 17.0.3 - Ran NiFi on Azul Zulu JDK 1.8.0-332 - NIFI-10086 Verified upgrade to Spring Framework 5.3.20 - NIFI-10096 Verified correct handling of inherited Parameter Contexts - NIFI-10098 Verified upgrade to Apache Tika 2.4.0 - NIFI-10114 Verified authorization configuration ShellUserGroupProvider - Ran NiFi Registry on Azul Zulu JDK 1.8.0-322 - Created Buckets - Ran NiFi Encrypt Config Toolkit with AES-GCM Thanks Joe! Regards, David Handermann On Mon, Jun 13, 2022 at 11:39 PM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.16.3. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1205 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.3/ > > A helpful reminder on how the release candidate verification process works: > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.16.3-RC1 > The Git commit ID is e15bdd7e3d09047d5fed70117b7c3dfd26f3a36e > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=e15bdd7e3d09047d5fed70117b7c3dfd26f3a36e > > Checksums of nifi-1.16.3-source-release.zip: > SHA256: c18edf739361246fe22bb4c2e5a4b1936b6199512b78638868f99b1d827b4d9e > SHA512: > > e3c9942737a0c2cf43fb3030da3cbee7d6be17038f0cf683c9522db25eb6d8664884594d5a7ce2183733568c06b9ccb52c3a6bf6d5ddcb334d2f84477cc68177 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 13 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351844 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.3 > > The vote will be open for 24 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. Then > please vote: > > [ ] +1 Release this package as nifi-1.16.3 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
CVE-2022-33140: Apache NiFi, Apache NiFi Registry: Improper Neutralization of Command Elements in Shell User Group Provider
Severity: high Description: The optional ShellUserGroupProvider in Apache NiFi 1.10.0 to 1.16.2 and Apache NiFi Registry 0.6.0 to 1.16.2 does not neutralize arguments for group resolution commands, allowing injection of operating system commands on Linux and macOS platforms. The ShellUserGroupProvider is not included in the default configuration. Command injection requires ShellUserGroupProvider to be one of the enabled User Group Providers in the Authorizers configuration. Command injection also requires an authenticated user with elevated privileges. Apache NiFi requires an authenticated user with authorization to modify access policies in order to execute the command. Apache NiFi Registry requires an authenticated user with authorization to read user groups in order to execute the command. The resolution removes command formatting based on user-provided arguments. This issue is being tracked as NIFI-10114 Mitigation: Disabling the ShellUserGroupProvider mitigates the vulnerability.
[DISCUSS] Strategy for Dropping Java 8 Support in NiFi 2.0
Team, With multiple major projects in the process of sunsetting support for Java 8, we should also prepare a timeline for removing Java 8 support from Apache NiFi and subprojects. BACKGROUND The Jetty project announced the end of community support for version 9 as of 2022-06-01 [1]. Although Jetty 9 is not end of life in terms of security updates, this is an important milestone as both NiFi and NiFi Registry leverage Jetty for the web application container. Jetty 10 requires Java 11 as the minimum version. The next major release of the Spring Framework will drop support for both Java 8 and 11, requiring Java 17 as the minimum version [2]. Other supporting components, such as OpenSAML, which enables SAML 2 integration, dropped support for Java 8 in OpenSAML 4 [3]. In order to continue maintaining a secure product, NiFi will also need to remove Java 8 support so that we can track dependency upgrades. NEXT STEPS In light of widespread deployment of Apache NiFi and subprojects, we need to prepare a timeline for transition. Although there have been various discussions on what should be included in the next major release, narrowing the focus to simply removing support for Java 8 provides the simplest path forward. Announcing removal of support for Java 8 should incorporate a reasonable amount of time for potential transition. NiFi has supported Java 11 for multiple releases, and NiFi 1.16.0 included basic support for Java 17. At minimum, it seems best to proceed with a release for NiFi 1.17.0, when ready, without making any changes. At that time, we should also have a timeline for removing Java 8 support. It may be worthwhile to plan on at least one more minor release that incorporates deprecation warnings where necessary. Following a selected minor release version, a support branch for major version 1 could be created, as a means of providing critical security and functional fixes. With a support branch created, main development could be transitioned to 2.0.0-SNAPSHOT. I defer to Joe Witt as the release manager for more thought around these particular details. Please provide your thoughts on the general process, and highlight particular areas of concern. Regards, David Handermann [1] https://github.com/eclipse/jetty.project/issues/7958 [2] https://spring.io/blog/2021/09/02/a-java-17-and-jakarta-ee-9-baseline-for-spring-framework-6 [3] https://shibboleth.atlassian.net/wiki/spaces/OSAML/overview
Re: [DISCUSS] Release for NAR Maven Plugin
Thanks Kevin, +1 for preparing a release of the NAR Maven plugin. Regards, David Handermann On Wed, Jun 15, 2022 at 11:58 AM Joe Witt wrote: > +1 > > On Wed, Jun 15, 2022 at 9:41 AM Kevin Doran wrote: > > > All, > > > > If there are no objections, I would like to put out a maintenance release > > (1.3.4) of the NAR Maven plugin, which has had some recent bug fixes and > > improvements: > > > > > >1. https://issues.apache.org/jira/browse/NIFI-10011 > >2. https://issues.apache.org/jira/browse/NIFI-9856 > >3. https://issues.apache.org/jira/browse/NIFI-9857 > > > > > > I would be glad to take the role of RM for this release. > > > > Thanks, > > Kevin > > >
Re: [DISCUSS] Strategy for Dropping Java 8 Support in NiFi 2.0
Thanks for the replies Kevin and Pierre! Various JDK vendors have different timelines for Java 11 support, some planning to end active support in September 2023 and others in October 2024. Either way, I agree that moving to Java 11 as the minimum version should be a shorter duration, with the goal of making Java 17 the minimum before too much time elapses. As far as a general timeline for removing Java 8 support in NiFi, a good goal in my mind would be no later than the end of this calendar year, 2022. Regards, David Handermann On Wed, Jun 15, 2022 at 11:55 AM Pierre Villard wrote: > I'd even love to go straight to Java 17 since it's the new LTS version... > but this may be quite a big jump for our community and users. > I guess we can envision a "short" 2.x release line and consider Java 17 for > 3.x. > Definitely approve the proposal! > > Le mer. 15 juin 2022 à 18:50, Kevin Doran a écrit : > > > Thanks for reviving this discussion David. In light of those core > > dependencies dropping support for Java 8, this plan seems necessary for > > NiFi. I support the proposal. > > > > Thanks, > > Kevin > > > > On Jun 15, 2022 at 11:48:05, David Handermann < > exceptionfact...@apache.org > > > > > wrote: > > > > > Team, > > > > > > With multiple major projects in the process of sunsetting support for > > Java > > > 8, we should also prepare a timeline for removing Java 8 support from > > > Apache NiFi and subprojects. > > > > > > BACKGROUND > > > > > > The Jetty project announced the end of community support for version 9 > as > > > of 2022-06-01 [1]. Although Jetty 9 is not end of life in terms of > > security > > > updates, this is an important milestone as both NiFi and NiFi Registry > > > leverage Jetty for the web application container. Jetty 10 requires > Java > > 11 > > > as the minimum version. > > > > > > The next major release of the Spring Framework will drop support for > both > > > Java 8 and 11, requiring Java 17 as the minimum version [2]. Other > > > supporting components, such as OpenSAML, which enables SAML 2 > > integration, > > > dropped support for Java 8 in OpenSAML 4 [3]. > > > > > > In order to continue maintaining a secure product, NiFi will also need > to > > > remove Java 8 support so that we can track dependency upgrades. > > > > > > NEXT STEPS > > > > > > In light of widespread deployment of Apache NiFi and subprojects, we > need > > > to prepare a timeline for transition. Although there have been various > > > discussions on what should be included in the next major release, > > narrowing > > > the focus to simply removing support for Java 8 provides the simplest > > path > > > forward. > > > > > > Announcing removal of support for Java 8 should incorporate a > reasonable > > > amount of time for potential transition. NiFi has supported Java 11 for > > > multiple releases, and NiFi 1.16.0 included basic support for Java 17. > > > > > > At minimum, it seems best to proceed with a release for NiFi 1.17.0, > when > > > ready, without making any changes. At that time, we should also have a > > > timeline for removing Java 8 support. It may be worthwhile to plan on > at > > > least one more minor release that incorporates deprecation warnings > where > > > necessary. > > > > > > Following a selected minor release version, a support branch for major > > > version 1 could be created, as a means of providing critical security > and > > > functional fixes. With a support branch created, main development could > > be > > > transitioned to 2.0.0-SNAPSHOT. I defer to Joe Witt as the release > > manager > > > for more thought around these particular details. > > > > > > Please provide your thoughts on the general process, and highlight > > > particular areas of concern. > > > > > > Regards, > > > David Handermann > > > > > > [1] https://github.com/eclipse/jetty.project/issues/7958 > > > [2] > > > > > > > > > https://spring.io/blog/2021/09/02/a-java-17-and-jakarta-ee-9-baseline-for-spring-framework-6 > > > [3] https://shibboleth.atlassian.net/wiki/spaces/OSAML/overview > > > > > >
Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.4
+1 (binding) - Verified signature and hashes - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-332 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 17.0.3 Thanks Kevin! Regards, David Handermann On Thu, Jun 16, 2022 at 4:11 PM Kevin Doran wrote: > Hello Apache NiFi Community, > > I am pleased to be calling this vote for the source release of Apache > NiFi NAR Maven Plugin 1.3.4. > > The source being voted upon, including signatures, digests, etc. can > be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.4/ > > The Git tag is nifi-nar-maven-plugin-1.3.4-RC1 > The Git commit ID is c4d9563bff2b5c2120e7f1e181343e2c01cc0422 > > https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=c4d9563bff2b5c2120e7f1e181343e2c01cc0422 > > Checksums of nifi-nar-maven-plugin-1.3.4-source-release.zip: > SHA256: acee55a767b9c90c8884e8c6e5fe936088243f778dc7e1de6f281cd4c8a85cab > SHA512: > 267cc157117d179517257f47c8d92f3dc5e594b230bb019ccc6e2cb2884b450af56603214fa651bc33aceb3bf0bdb9369aa779672cb3f6f0efd2bacb166f9051 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/kdoran.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 3 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350909 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.4 > > The vote will be open for 72 hours. Please download the release > candidate and evaluate the necessary items including checking hashes, > signatures, build from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-nar-maven-plugin-1.3.4 > [ ] +0 no opinion > [ ] -1 Do not release this package because ... >
Re: [DISCUSS] Strategy for Dropping Java 8 Support in NiFi 2.0
Team, I created the following Jira issue to capture the details of this discussion: https://issues.apache.org/jira/browse/NIFI-10147 We can continue the discussion on this thread, and include more specific comments on the Jira issue as well. As mentioned in the Jira issue, we should revisit a more definite implementation plan following the release of NiFi 1.17.0. Regards, David Handermann On Wed, Jun 15, 2022 at 12:31 PM Kevin Doran wrote: > Pierre and David, I agree with this project goals: > > >- a 2.x release that drops support for Java 8 (requires at least Java >11) by EOY >- a 3.x release that drops support for Java 11 (requires at least Java >17) in the not-to-distant future, perhaps 2023/24 > > > This would also mean we could move some of the original goals of 2.x to > target the 3.x line instead, given the deadlines David identified. > > Kevin > > On Jun 15, 2022 at 13:20:41, David Handermann > > wrote: > > > Thanks for the replies Kevin and Pierre! > > > > Various JDK vendors have different timelines for Java 11 support, some > > planning to end active support in September 2023 and others in October > > 2024. Either way, I agree that moving to Java 11 as the minimum version > > should be a shorter duration, with the goal of making Java 17 the minimum > > before too much time elapses. > > > > As far as a general timeline for removing Java 8 support in NiFi, a good > > goal in my mind would be no later than the end of this calendar year, > 2022. > > > > Regards, > > David Handermann > > > > On Wed, Jun 15, 2022 at 11:55 AM Pierre Villard < > > pierre.villard...@gmail.com> > > wrote: > > > > I'd even love to go straight to Java 17 since it's the new LTS version... > > > > but this may be quite a big jump for our community and users. > > > > I guess we can envision a "short" 2.x release line and consider Java 17 > for > > > > 3.x. > > > > Definitely approve the proposal! > > > > > > Le mer. 15 juin 2022 à 18:50, Kevin Doran a écrit : > > > > > > > Thanks for reviving this discussion David. In light of those core > > > > > dependencies dropping support for Java 8, this plan seems necessary for > > > > > NiFi. I support the proposal. > > > > > > > > > > Thanks, > > > > > Kevin > > > > > > > > > > On Jun 15, 2022 at 11:48:05, David Handermann < > > > > exceptionfact...@apache.org > > > > > > > > > > > wrote: > > > > > > > > > > > Team, > > > > > > > > > > > > With multiple major projects in the process of sunsetting support for > > > > > Java > > > > > > 8, we should also prepare a timeline for removing Java 8 support from > > > > > > Apache NiFi and subprojects. > > > > > > > > > > > > BACKGROUND > > > > > > > > > > > > The Jetty project announced the end of community support for version > 9 > > > > as > > > > > > of 2022-06-01 [1]. Although Jetty 9 is not end of life in terms of > > > > > security > > > > > > updates, this is an important milestone as both NiFi and NiFi > Registry > > > > > > leverage Jetty for the web application container. Jetty 10 requires > > > > Java > > > > > 11 > > > > > > as the minimum version. > > > > > > > > > > > > The next major release of the Spring Framework will drop support for > > > > both > > > > > > Java 8 and 11, requiring Java 17 as the minimum version [2]. Other > > > > > > supporting components, such as OpenSAML, which enables SAML 2 > > > > > integration, > > > > > > dropped support for Java 8 in OpenSAML 4 [3]. > > > > > > > > > > > > In order to continue maintaining a secure product, NiFi will also > need > > > > to > > > > > > remove Java 8 support so that we can track dependency upgrades. > > > > > > > > > > > > NEXT STEPS > > > > > > > > > > > > In light of widespread deployment of Apache NiFi and subprojects, we > > > > need > > > > > > to prepare a timeline for transition. Although there have been > various > > > > > > discussions on what should be included in the next
Re: Hashicorp vault provider for SSLContextServices
Hi Cannon, Understanding the SSLContextService [1] interface and the StandardSSLContextService [2] implementation would be a good starting point for considering a custom implementation. The most important interface method is createContext(), which returns an instance of javax.net.ssl.SSLContext [3] based on configured properties. The SSLContextService has other methods for returning configured values, but most components use the createContext() method. Certain components may use the createTlsConfiguration() method, or other methods to retrieve specific values. Those other methods could be problematic to implement when attempting to develop a service that is not based on files. Throwing an UnsupportedOperationException may be one option, which would result in runtime errors for components that do not use the standard createContext() method. Interacting with HashiCorp Vault involves its own set of configuration options depending on how it is deployed, but understanding the file-based approach in the StandardSSLContextService should provide a helpful background. Regards, David Handermann [1] https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-service-api/src/main/java/org/apache/nifi/ssl/SSLContextService.java [2] https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service/src/main/java/org/apache/nifi/ssl/StandardSSLContextService.java [3] https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLContext.html On Tue, Jul 12, 2022 at 2:13 PM Cannon Palms wrote: > We need to service up dynamic certificates to NiFi at runtime to enable > connection to remote hosts via certificate-based TLS. > > My understanding is that the existing implementations of SSLContextService > all require that the keystore/truststore be accessible through the > filesystem. > > What might an implementation based on third-party providers (namely vault) > look like? Can someone point me to any resources that might guide this > implementation? > > Thanks, > Cannon >
Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.5
+1 (binding) - Verified signatures and hashes - Ran build of nifi-nar-maven-plugin on Ubuntu Linux 22.04 with Azul Zulu JDK 1.8.0-332 - Ran build of NiFi using nifi-nar-maven-plugin 1.3.5 - Verified updates to nifi-runtime-manifests.json Thanks Bryan! Regards, David Handermann On Wed, Jul 20, 2022 at 3:37 PM Bryan Bende wrote: > Hello Apache NiFi Community, > > I am pleased to be calling this vote for the source release of Apache > NiFi NAR Maven Plugin 1.3.5. > > The source being voted upon, including signatures, digests, etc. can > be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.5/ > > The Git tag is nifi-nar-maven-plugin-1.3.5-RC1 > The Git commit ID is 77f516a65d796b715278bedd06407d4a8f98f771 > > https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=77f516a65d796b715278bedd06407d4a8f98f771 > > Checksums of nifi-nar-maven-plugin-1.3.5-source-release.zip: > SHA256: 073271bbd28b3d0b9b74c42ab889b4e7301e52b9a0579a3484e7ad0b7f8d3532 > SHA512: > 9fcc487a197a8d3c5cd297696eb303015c9cde32f89541682dfac921e862c1f1ee8ba5e41573eeb7a5b5190b95c3b49443fca9ed5d5a3f7768a4ea1b031829af > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/bbende.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 2 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351862 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.5 > > The vote will be open for 24 hours. Please download the release > candidate and evaluate the necessary items including checking hashes, > signatures, build from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-nar-maven-plugin-1.3.5 > [ ] +0 no opinion > [ ] -1 Do not release this package because ... >
Re: [discuss] nifi 1.17
Joe, Thanks for initiating the discussion! Moving forward with a release of 1.17.0 sounds good, pending the resolution of the issues currently flagged as in progress [1]. Thanks for referencing the recent discussion on a release timeline to remove support for Java 8 [2]. Keeping the general goal of dropping Java 8 support by the end of 2022, moving forward with 1.17.0 sounds good. Then we could consider deprecating Java 8 support for a release of 1.18.0. Based on the general timeline of recent releases, that should put things in a good position for a 2.0.0 release around the end of the year. We can discuss the particulars in a separate thread. Regards, David Handermann [1] https://issues.apache.org/jira/projects/NIFI/versions/12351438 [2] https://lists.apache.org/thread/mm1xf3b9nvrcgytb92oy3swvvc45fl34 On Sat, Jul 23, 2022 at 8:18 AM Joe Witt wrote: > Mike > > Needs some work (unit test) but would kick out RC without it. > > Thanks > > On Sat, Jul 23, 2022 at 6:12 AM Mike Thomsen > wrote: > > > I would put this one on the list of "potential blockers" for 1.17: > > https://github.com/apache/nifi/pull/6226 > > > > On Fri, Jul 22, 2022 at 4:52 PM Joe Witt wrote: > > > > > > Hello Everyone, > > > > > > Somehow Apache NiFi 1.16 is already four months old and we've landed > just > > > under 300 JIRAs [1] on the 1.17 line. So certainly time and more than > > > enough oomph in place to warrant a 1.17 release. I have seen quite a > few > > > asks lately about when we'll do it. So looking at things in JIRA it > > seems > > > pretty well buttoned up. A few bugs being worked out but with PRs. > > > > > > Following this cool discussion [2] we now have the ability to build > NiFi > > on > > > ARM/latest Macbooks which is super helpful for many in the community as > > > well. > > > > > > I'm happy to do the RM task. Will take a look at these few bug PRs on > > > Monday and if we're looking good will kick off RC1. > > > > > > [1] > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351438 > > > [2] https://lists.apache.org/thread/mm1xf3b9nvrcgytb92oy3swvvc45fl34 > > > > > > Thanks > > > Joe > > >
Re: [VOTE] Release Apache NiFi 1.17.0 (rc1)
-1 (binding) After a number of positive verifications across multiple features, I found the following issues: - https://issues.apache.org/jira/browse/NIFI-10284 HTTP Request Log Missing Authenticated User - https://issues.apache.org/jira/browse/NIFI-10286 C2 API libraries present in root library directory I submitted the following pull requests to address the issues: - NIFI-10284 https://github.com/apache/nifi/pull/6250 - NIFI-10286 https://github.com/apache/nifi/pull/6251 Regards, David Handermann On Mon, Jul 25, 2022 at 9:57 PM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.17.0. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1209 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.17.0/ > > A helpful reminder on how the release candidate verification process works: > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.17.0-RC1 > The Git commit ID is cfa99de02ab5a440e94a2bb0044822ae9d0d99e7 > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=cfa99de02ab5a440e94a2bb0044822ae9d0d99e7 > > Checksums of nifi-1.17.0-source-release.zip: > SHA256: 060ac40bbd6f3bac08f724bf040a68d52f4449d2e529af9c1d61db8932aa7b1b > SHA512: > > 97440184d0c34e1f287eba648eb47e5c276df46d6d89c376616939380d95afb6c03b4e4e9ea4170ba62d300637387d532314381225d000d529a6cc5bc4e8b436 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 305 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351438 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.17.0 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. Then > please vote: > > [ ] +1 Release this package as nifi-1.17.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
Re: [VOTE] Release Apache NiFi 1.17.0 (RC2)
+1 (binding) - Verified signatures and hashes - Ran build using Maven 3.8.6 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-332 AMD64 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 11.0.12 AMD64 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 17.0.4 AMD64 - Ran build on macOS 12.5 with Azul Zulu JDK 17.0.4 AArch64 - Ran system tests on Azul Zulu JDK 17.0.4 AArch64 - Ran stateless system tests on Azul Zulu JDK 17.0.4 Aarch64 - Ran NiFi on Azul Zulu JDK 17.0.4 AArch64 and Azul Zulu JDK 1.8.0-332 AMD64 - Configured Repository Encryption - Ran NiFi Registry on Azul Zulu JDK 17.0.4 AArch64 - Created Buckets - Verified Flow Version Control - Ran NiFi Encrypt Config Toolkit with AES-GCM - Ran MiNiFi on Azul Zulu JDK 17.0.4 AMD64 - NIFI-3869 Verified ListenHTTP and HandleHttpRequest support for HTTP/2 - NIFI-9804 Verified Application Server configuration support for HTTP/2 - NIFI-9958 Verified persistence of Sensitive Dynamic Properties - NIFI-9959 Verified UI handling of support for Sensitive Dynamic Properties - NIFI-9961 Verified InvokeHTTP support for Sensitive Dynamic Properties - NIFI-9962 Verified DBCPConnectionPool support for Sensitive Dynamic Properties - NIFI-9975 Verified ConsumeTwitter Sample Stream - NIFI-9838 Verified ListenTCPRecord adds client certificate FlowFile attributes - NIFI-9849 Verified SAML authentication with Keycloak and Single Logout enabled - NIFI-10044 Verified YAML formatted content viewer - NIFI-10010 Verified ListenTCP adds client certificate FlowFile attributes - NIFI-10069 Verified ExecuteStreamCommand support for Sensitive Dynamic Properties - NIFI-10087 Verified UDPEventRecordSink configured in PutRecord sending to ListenUDPRecord - NIFI-10161 Verified InvokeHTTP and ListenHTTP support for Gzip Content-Encoding - NIFI-10165 Verified successful HTTP notification for bootstrap events - NIFI-10196 Verified JoltTransformJSON Advanced UI authentication handling - NIFI-10206 Verified correct HTTP processing in HandleHttpResponse - NIFI-10284 Verified authenticated username logged in nifi-request.log - NIFI-10286 Verified no C2 libraries present in root library directory Thanks Joe! Regards, David Handermann On Thu, Jul 28, 2022 at 9:28 AM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.17.0. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1210 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.17.0/ > > A helpful reminder on how the release candidate verification process works: > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.17.0-RC2 > The Git commit ID is 8d256784d84cc28bf5642e1bf38dec3eba0c5f23 > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=8d256784d84cc28bf5642e1bf38dec3eba0c5f23 > > Checksums of nifi-1.17.0-source-release.zip: > SHA256: 8b9b2088ad966329248cfae7792f576f4f30fea4b4e50f055f84724dba4fe8a3 > SHA512: > > 2429348ad514ca7ab9df86aaa57207f1434044c6f7e947d0950ca9826b4f1aa51061617a17444c086eed19b1f26a5ebbe3b455cafed9d219727adf26ecb5f8d2 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 310 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351438 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.17.0 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. Then > please vote: > > [ ] +1 Release this package as nifi-1.17.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
New NiFi PMC Member Nathan Gough
Apache NiFi Community, On behalf of the Apache NiFi Project Management Committee, I am very pleased to announce that Nathan Gough has accepted the invitation to join the PMC. Nathan has been a consistent contributor to the project since 2018. In addition to developing features and fixes, he regularly reviews pull requests and assists others through mailing lists and Slack conversations. He also plays an active role in reviewing and updating security reports for the project. Please join us in congratulating and welcoming Nathan to the Apache NiFi PMC. Congratulations Nathan!
Re: Duplicated dependencies
Dan, Thanks for asking about these dependencies. The attached image did not come through to the list, but the XML snippet shows the standard test dependencies. These dependencies are applied to all modules by design, to avoid unnecessary repetition in child modules. Regards, David Handermann On Thu, Sep 29, 2022 at 4:06 PM Dan S wrote: > When looking in the top level pom.xml in IntelliJ I see > > [image: image.png] > > > The duplicate dependencies are: > > org.junit.jupiter > junit-jupiter-api > test > > > org.junit.jupiter > junit-jupiter-engine > test > > > org.mockito > mockito-core > test > > > org.slf4j > slf4j-simple > test > > > org.codehaus.groovy > groovy-test > test > > > I assume these are oversites. Is this something a ticket should be made > for to fix? > > > >
Re: Duplicated dependencies
Dan, Thanks for the text reference and clarification. With those details, yes, it looks like the duplicates should be removed from the modules listed. Feel free to create a Jira issue to address this, and we should also consider implementing some enforcement to avoid future duplicates. Regards, David Handermann On Thu, Sep 29, 2022 at 4:21 PM Dan S wrote: > Sorry about that. I did not realize I could not include an image. Here is a > text version: > > Warning:(624, 10) Dependency is duplicated in file(s): > nifi-zendesk-processors > Warning:(629, 10) Dependency is duplicated in file(s): > nifi-zendesk-processors > Warning:(644, 10) Dependency is duplicated in file(s): > minifi-c2-provider-cache, minifi-c2-provider-delegating, > minifi-c2-provider-nifi-rest, minifi-c2-provider-util, > nifi-azure-reporting-task, nifi-pgp-processors, nifi-pgp-service, > nifi-registry-framework, nifi-registry-jetty, > nifi-registry-revision-spring-jdbc, nifi-registry-aws-extensions, > nifi-registry-ranger-plugin, and nifi-registry-toolkit-persistence > Warning:(654, 10) Dependency is duplicated in file(s): > nifi-system-test-suite > Warning:(659, 10) Dependency is duplicated in file(s): > nifi-registry-framework, nifi-registry-jetty, nifi-registry-properties, > nifi-registry-security-utils, and nifi-registry-web-api > > On Thu, Sep 29, 2022 at 5:16 PM David Handermann < > exceptionfact...@apache.org> wrote: > > > Dan, > > > > Thanks for asking about these dependencies. The attached image did not > come > > through to the list, but the XML snippet shows the standard test > > dependencies. These dependencies are applied to all modules by design, to > > avoid unnecessary repetition in child modules. > > > > Regards, > > David Handermann > > > > On Thu, Sep 29, 2022 at 4:06 PM Dan S wrote: > > > > > When looking in the top level pom.xml in IntelliJ I see > > > > > > [image: image.png] > > > > > > > > > The duplicate dependencies are: > > > > > > org.junit.jupiter > > > junit-jupiter-api > > > test > > > > > > > > > org.junit.jupiter > > > junit-jupiter-engine > > > test > > > > > > > > > org.mockito > > > mockito-core > > > test > > > > > > > > > org.slf4j > > > slf4j-simple > > > test > > > > > > > > > org.codehaus.groovy > > > groovy-test > > > test > > > > > > > > > I assume these are oversites. Is this something a ticket should be made > > > for to fix? > > > > > > > > > > > > > > >
Re: Duplicated dependencies
Dan, The initial removal of duplicate dependencies sounds good on its own. Enforcing the rules might take a bit more evaluation, so having it as a separate issue seems like the best way forward. Regards, David Handermann On Thu, Sep 29, 2022 at 5:11 PM Dan S wrote: > Should the ticket include in addition to the removal of the duplicate > dependencies the enforcement rules or just focus on the removal of the > duplicate dependencies? > > On Thu, Sep 29, 2022 at 5:56 PM David Handermann < > exceptionfact...@apache.org> wrote: > > > Dan, > > > > Thanks for the text reference and clarification. With those details, yes, > > it looks like the duplicates should be removed from the modules listed. > > Feel free to create a Jira issue to address this, and we should also > > consider implementing some enforcement to avoid future duplicates. > > > > Regards, > > David Handermann > > > > On Thu, Sep 29, 2022 at 4:21 PM Dan S wrote: > > > > > Sorry about that. I did not realize I could not include an image. Here > > is a > > > text version: > > > > > > Warning:(624, 10) Dependency is duplicated in file(s): > > > nifi-zendesk-processors > > > Warning:(629, 10) Dependency is duplicated in file(s): > > > nifi-zendesk-processors > > > Warning:(644, 10) Dependency is duplicated in file(s): > > > minifi-c2-provider-cache, minifi-c2-provider-delegating, > > > minifi-c2-provider-nifi-rest, minifi-c2-provider-util, > > > nifi-azure-reporting-task, nifi-pgp-processors, nifi-pgp-service, > > > nifi-registry-framework, nifi-registry-jetty, > > > nifi-registry-revision-spring-jdbc, nifi-registry-aws-extensions, > > > nifi-registry-ranger-plugin, and nifi-registry-toolkit-persistence > > > Warning:(654, 10) Dependency is duplicated in file(s): > > > nifi-system-test-suite > > > Warning:(659, 10) Dependency is duplicated in file(s): > > > nifi-registry-framework, nifi-registry-jetty, nifi-registry-properties, > > > nifi-registry-security-utils, and nifi-registry-web-api > > > > > > On Thu, Sep 29, 2022 at 5:16 PM David Handermann < > > > exceptionfact...@apache.org> wrote: > > > > > > > Dan, > > > > > > > > Thanks for asking about these dependencies. The attached image did > not > > > come > > > > through to the list, but the XML snippet shows the standard test > > > > dependencies. These dependencies are applied to all modules by > design, > > to > > > > avoid unnecessary repetition in child modules. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Thu, Sep 29, 2022 at 4:06 PM Dan S wrote: > > > > > > > > > When looking in the top level pom.xml in IntelliJ I see > > > > > > > > > > [image: image.png] > > > > > > > > > > > > > > > The duplicate dependencies are: > > > > > > > > > > org.junit.jupiter > > > > > junit-jupiter-api > > > > > test > > > > > > > > > > > > > > > org.junit.jupiter > > > > > junit-jupiter-engine > > > > > test > > > > > > > > > > > > > > > org.mockito > > > > > mockito-core > > > > > test > > > > > > > > > > > > > > > org.slf4j > > > > > slf4j-simple > > > > > test > > > > > > > > > > > > > > > org.codehaus.groovy > > > > > groovy-test > > > > > test > > > > > > > > > > > > > > > I assume these are oversites. Is this something a ticket should be > > made > > > > > for to fix? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Release Apache NiFi 1.18.0 (RC3)
+1 (binding) - Verified signatures and hashes - Ran build using Maven 3.8.6 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-345 AMD64 - Ran build on macOS 12.6 with Azul Zulu JDK 17.0.4 AArch64 - Ran stateless and system tests on macOS 12.6 with Azul Zulu JDK 17.0.4 AArch64 - Ran NiFi on macOS 12.6 with Azul Zulu JDK 17.0.4 AArch64 - Ran NiFi on macOS 12.5 with Azul Zulu JDK 11.0.16 AMD64 - Ran NiFi on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-345 AMD64 - Ran NiFi on Windows 11 with Azul Zulu JDK 17.0.4 AMD64 - Configured and ran with Repository Encryption - Configured and ran with SAML 2 authentication and Nginx reverse proxy - Configured and ran with OpenID Connect authentication and Nginx reverse proxy - Ran NiFi Registry on macOS 12.6 with Azul Zulu JDK 17.0.4 AArch64 - Created Buckets - Verified Flow Version Control - NIFI-3964 Verified GrokReader support for other Resource Types in Grok Patterns property - NIFI-8648 Verified Session Affinity documentation section in Administrator's Guide - NIFI-9374 Verified nifi-deprecation.log configuration and NiFi Legacy Cipher Provider warnings - NIFI-9401 Verified configuration of HashiCorp Vault Parameter Provider - NIFI-9500 Verified nifi.cmd command on Windows 11 with set-single-user-credentials - NIFI-10300 Verified sending events with PutRecord and AzureEventHubRecordSink - NIFI-10313 Verified expected Session Expired message and link after token expiration - NIFI-10318 Verified correct handling of Client Authentication in HandleHttpRequest - NIFI-10322 Verified successful OIDC authentication and Session Expired message when running with Nginx reverse proxy and Keycloak - NIFI-10381 Verified Azure SDK upgrade with GetAzureEventHub - NIFI-10435 Verified Sensitive Dynamic Property value arguments masked in - ExecuteStreamCommand - NIFI-10471 Verified InvokeHTTP Proxy Host property deprecation logging - NIFI-10497 Verified Registry Client integration with versioned Flows - NIFI-10505 Verified encryption of application properties with HashiCorp Vault Property Provider - NIFI-10559 Verified correct Remote Path documentation in PutSFTP Thanks Joe! Regards, David Handermann On Thu, Sep 29, 2022 at 2:22 PM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache > NiFi 1.18.0. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1213 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.18.0/ > > A helpful reminder on how the release candidate verification process works: > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.18.0-RC3 > The Git commit ID is 5bc64c812b2c76ee2879d8081ceadf62d5e3c702 > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=5bc64c812b2c76ee2879d8081ceadf62d5e3c702 > > Checksums of nifi-1.18.0-source-release.zip: > SHA256: bd1b675f17dbf712089a79f7bc043eae2df63bcc2e08b2012a6431641037679f > SHA512: > cea43af57089128ff65bb53e76b2fdfa8dec7397e2bf45d41e35b758b731355075839b9c018ee6284cb15e293b105e248d88748148960ad80ae387824139f52b > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 171 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12352150 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.18.0 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build > from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-1.18.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
Re: [VOTE] Release Apache NiFi 1.18.0 (RC4)
+1 (binding) - Verified signatures and hashes - Ran build using Maven 3.8.6 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-345 AMD64 - Ran build on macOS 12.6 with Azul Zulu JDK 17.0.4 AArch64 - Ran stateless and system tests on macOS 12.6 with Azul Zulu JDK 17.0.4 AArch64 - Ran NiFi on macOS 12.6 with Azul Zulu JDK 17.0.4 AArch64 - Ran NiFi on macOS 12.6 with Azul Zulu JDK 11.0.16 x86_64 - Ran NiFi on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-345 AMD64 - Ran NiFi on Windows 11 with Azul Zulu JDK 17.0.4 AMD64 - Configured and ran with Repository Encryption - Configured AES-GCM encryption of sensitive properties - Ran NiFi Registry on Ubuntu 22.04 with Azul Zulu JDK 11.0.12 AMD64 - Created Buckets - Verified Flow Version Control - Ran NiFi Stateless on macOS 12.6 with Azul Zulu JDK 11.0.16 x86_64 Thanks Joe! Regards, David Handermann On Mon, Oct 3, 2022 at 3:45 PM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache > NiFi 1.18.0. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1214 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.18.0/ > > A helpful reminder on how the release candidate verification process works: > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.18.0-RC4 > The Git commit ID is 109e54cd585902a981d1b370b3dc4d1620be438c > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=109e54cd585902a981d1b370b3dc4d1620be438c > > Checksums of nifi-1.18.0-source-release.zip: > SHA256: 925cbb92c107d0fa3194a349d985cff4933a61b2555eff57b1b81433fe37c139 > SHA512: > f143215b1746342e7584f5ad65b546fcc378cd78aa17628fb605dfdbbaf11e897a0173dd67807fc90cb18c17124a4227d5fe07e7ed609d9ed1904503b757c604 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 184 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12352150 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.18.0 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build > from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-1.18.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
Re: Suggestion for a new Expression Language method
Hi Dan, It seems like this could be helpful, although accurate detection might be difficult. It could be as simple as checking whether a string starts with a curly bracket or square bracket character. Another option would be attempting to parse the string using a JSON library, since that could cover additional bases. Either way, having sufficient unit tests and documentation would be helpful depending on the implementation approach. Regards, David Handermann On Fri, Oct 7, 2022 at 10:06 AM Pierre Villard wrote: > Hi Dan, > > I personally think this is a fair addition to the expression language > options we have. Feel free to file a JIRA and submit a pull request if > you'd like to do so. > > Thanks, > Pierre > > Le ven. 7 oct. 2022 à 08:34, Dan S a écrit : > > > My team has flow file attributes which we do not always know whether they > > are JSON or not. We would like to run JsonPath on them but when they are > > not JSON we get flow files yielding as detailed in NIFI-10396 > > <https://issues.apache.org/jira/browse/NIFI-10396>. I understand the > > reasons given there not to support the patch but I was thinking of > possibly > > another solution which could short circuit the issue. I wanted to > propose a > > new Expression Language method isJson which would determine whether an > > attribute is JSON or not. This would allow for determining whether an > > attribute is JSON and pass it along to JsonPath or route accordingly when > > it is not JSON. Does that sound reasonable? > > >
Re: Suggestion for a new Expression Language method
Dan, That comment appears to be historical. The current ObjectMapper documentation indicates that instances are thread safe as long as all configuration is completed before read or write operations: https://github.com/FasterXML/jackson-databind/blob/2.14/src/main/java/com/fasterxml/jackson/databind/ObjectMapper.java#L81 Regards, David Handermann On Fri, Oct 7, 2022 at 11:05 AM Dan S wrote: > Mike, > I noticed you did similar work in NIFI-5271 > <https://issues.apache.org/jira/browse/NIFI-5271>. Joe Witt commented > there > > > We dont have guaranteed thread safety until the jackson 3.0 release. > > > Has that changed? > > On Fri, Oct 7, 2022 at 11:55 AM Mike Thomsen > wrote: > > > To do it right, you want to use a tool like Jackson or Gson to do a > > test parse after you've detected open and close statements that seem > > to be JSON open and close statements. > > > > On Fri, Oct 7, 2022 at 11:38 AM Dan S wrote: > > > > > > Mike, > > > What do you mean by "this might be a somewhat heavy method so be > > careful." > > > ? > > >
Re: SSLContext from PEM files
Mike, Thanks for raising this issue, can you provide some links to the documentation and source code for Neo4j? Although the SSL Context Service supports direct access to the Keystore and Trust Store properties, most use cases involve having the service instantiate an SSLContext. In this particular case, it may be better to specify those properties directly in a Neo4j component, as opposed to having an SSL Context Service that is essentially passing through property values. Those are a couple initial thoughts, having some additional background would help evaluate the best approach. Regards, David Handermann On Tue, Oct 11, 2022 at 12:36 PM Mike Thomsen wrote: > Neo4J for some reason doesn't support the standard Java keystore types > or P12 files for its client SSL configuration. It requires the use of > PEM files. Would it be better to extend the SSLContext service types > to include support for PEM files or create an all new SSL Provider > type that is geared toward only reading from PEM files? > > Thanks, > > Mike >
