Re: [discuss] NiFi 1.20 and NiFi 2.0
I think this is a good, practical discussion. On the one hand, we can't put off 2.x any longer as we really need to updated the minimum required Java to 11. Updating main development to target 2.x feels like a good way drive progress on that along with the other 2.0 goals. On the other hand, the concerns are valid: moving all development to target 2.x puts the project at risk if we cannot release 2.0.0 on a reasonable timeline. The restricted scope of 2.0 helps, but this stated release goal feels risky to me: Implement Migration Tools for Upgrading Flows - Implement automated migration where possible to remap properties and features - Implement migration tools for manual conversion of XML Templates to JSON Flow Definitions - Create documentation for manual steps necessary where programmatic migration cannot be implemented - NiFi 2.0 should be capable of starting with ghosted components for removed Processors or Controller Services Removing deprecated components should be fairly straightforward and quick, but automating and documenting migration is a large effort. On this po On Jan 10, 2023 at 09:32:31, Bryan Bende wrote: > The plan as I understand it is not to diverge and create separate feature > development on the 1.x line, so I would expect all PRs to continue to be > submitted only to main. We would release 1.x as needed with major bug fixes > or critical security updates, and these would be cherry-picked and/or > backported as necessary, mostly without the need for PRs, the same as we > would do if we were bringing fixes from main (1.20.0-SNAPSHOT) back to a > maintenance line like (1.19.x). For precedent, we followed this same > approach going from the 0.x line to 1.0.0 and there wasn't any major issue. > > On Tue, Jan 10, 2023 at 7:07 AM Otto Fowler > wrote: > > It was also mentioned in another thread that we need to have agreement on > > our explicit strategy and support for 1.x going forward, did that happen? > > > From: Otto Fowler > > Reply: Otto Fowler > > Date: January 10, 2023 at 07:02:34 > > To: dev@nifi.apache.org > > Subject: Re: [discuss] NiFi 1.20 and NiFi 2.0 > > > There needs to be an update to the contributing guide as to how to submit > > PRs to 1.x or 2.x etc. > > > From: Joe Witt > > Reply: dev@nifi.apache.org > > Date: January 9, 2023 at 15:53:16 > > To: dev@nifi.apache.org > > Subject: [discuss] NiFi 1.20 and NiFi 2.0 > > > Team, > > > As David mentioned in [1] following a successful NiFi 2.0 release goal > > planning - we are now going to start moving the 'main' line to be the NiFi > > 2.0 line which will allow for the key work to take place. We will also > > move niFi 1.x to its appropriate support line. > > > It is also the case that we have nearly 100 JIRAs on NiFi 1.20 and we have > > work in there including security items so it is time to make a release. > > The intent then is to initiate 1.20 and immediate after that change 'main' > > to 2.0. > > > Going forward then all work on the 1.x line should be focused on > > maintaining existing features, dependencies, and helping 1.x users migrate > > to the 2.x line. Otherwise, new feature work will happen on 'main' as it > > normally does and will come out in the NiFi 2.x release line. > > > Please flag key outstanding items as we narrow down the release candidate > > for NiFi 1.20. > > > Thanks > > Joe > > > [1] https://lists.apache.org/thread/qo4vvdw46235y7vy2crcd6l4m11wl7jz > > >
Re: [discuss] NiFi 1.20 and NiFi 2.0
[hit the wrong keyboard shortcut, here is the rest of my thoughts] On this point from David: We may need to have a longer release candidate period, or more incremental > fix releases > for the initial 2.0.0 release train, but I do not expect delaying a 2.0.0 > release for new features, as that is not part of the release goals. > Would the community benefit from one or more milestone releases of 2.0, to allow for a wider group to run / live on the proposed 2.0 prior to releasing it as "stable"? I know we've never done a milestone release in the past, and I'm not sure what ASF guidance is on the topic, but if it could be beneficial we could look into that. Cheers, Kevin On Jan 11, 2023 at 12:22:43, Kevin Doran wrote: > I think this is a good, practical discussion. > > On the one hand, we can't put off 2.x any longer as we really need to > updated the minimum required Java to 11. Updating main development to > target 2.x feels like a good way drive progress on that along with the > other 2.0 goals. > > On the other hand, the concerns are valid: moving all development to > target 2.x puts the project at risk if we cannot release 2.0.0 on a > reasonable timeline. The restricted scope of 2.0 helps, but this stated > release goal feels risky to me: > > Implement Migration Tools for Upgrading Flows > > >- Implement automated migration where possible to remap properties and > features > - Implement migration tools for manual conversion of XML Templates > to JSON Flow Definitions > - Create documentation for manual steps necessary where > programmatic migration cannot be implemented > - NiFi 2.0 should be capable of starting with ghosted components > for removed Processors or Controller Services > > > Removing deprecated components should be fairly straightforward and quick, > but automating and documenting migration is a large effort. > > On this po > > > On Jan 10, 2023 at 09:32:31, Bryan Bende wrote: > >> The plan as I understand it is not to diverge and create separate feature >> development on the 1.x line, so I would expect all PRs to continue to be >> submitted only to main. We would release 1.x as needed with major bug >> fixes >> or critical security updates, and these would be cherry-picked and/or >> backported as necessary, mostly without the need for PRs, the same as we >> would do if we were bringing fixes from main (1.20.0-SNAPSHOT) back to a >> maintenance line like (1.19.x). For precedent, we followed this same >> approach going from the 0.x line to 1.0.0 and there wasn't any major >> issue. >> >> On Tue, Jan 10, 2023 at 7:07 AM Otto Fowler >> wrote: >> >> It was also mentioned in another thread that we need to have agreement on >> >> our explicit strategy and support for 1.x going forward, did that happen? >> >> >> From: Otto Fowler >> >> Reply: Otto Fowler >> >> Date: January 10, 2023 at 07:02:34 >> >> To: dev@nifi.apache.org >> >> Subject: Re: [discuss] NiFi 1.20 and NiFi 2.0 >> >> >> There needs to be an update to the contributing guide as to how to submit >> >> PRs to 1.x or 2.x etc. >> >> >> From: Joe Witt >> >> Reply: dev@nifi.apache.org >> >> Date: January 9, 2023 at 15:53:16 >> >> To: dev@nifi.apache.org >> >> Subject: [discuss] NiFi 1.20 and NiFi 2.0 >> >> >> Team, >> >> >> As David mentioned in [1] following a successful NiFi 2.0 release goal >> >> planning - we are now going to start moving the 'main' line to be the NiFi >> >> 2.0 line which will allow for the key work to take place. We will also >> >> move niFi 1.x to its appropriate support line. >> >> >> It is also the case that we have nearly 100 JIRAs on NiFi 1.20 and we have >> >> work in there including security items so it is time to make a release. >> >> The intent then is to initiate 1.20 and immediate after that change 'main' >> >> to 2.0. >> >> >> Going forward then all work on the 1.x line should be focused on >> >> maintaining existing features, dependencies, and helping 1.x users migrate >> >> to the 2.x line. Otherwise, new feature work will happen on 'main' as it >> >> normally does and will come out in the NiFi 2.x release line. >> >> >> Please flag key outstanding items as we narrow down the release candidate >> >> for NiFi 1.20. >> >> >> Thanks >> >> Joe >> >> >> [1] https://lists.apache.org/thread/qo4vvdw46235y7vy2crcd6l4m11wl7jz >> >> >>
Re: [discuss] NiFi 1.20 and NiFi 2.0
Kevin, Yeah we can do whatever we want as far as 'releases' of 2.0 that are prior to us officially considering it 2.0/stable. That said - the migration tooling will be key to provide as we need to make the bridge as solid and stable as possible to help someone move from 1.x to 2.x. I dont know how related these two concepts (milestone releases and 1.x to 2.x ease really are). Thanks On Wed, Jan 11, 2023 at 10:27 AM Kevin Doran wrote: > [hit the wrong keyboard shortcut, here is the rest of my thoughts] > > On this point from David: > > We may need to have a longer release candidate period, or more incremental > > fix releases > > for the initial 2.0.0 release train, but I do not expect delaying a 2.0.0 > > release for new features, as that is not part of the release goals. > > > > Would the community benefit from one or more milestone releases of 2.0, to > allow for a wider group to run / live on the proposed 2.0 prior to > releasing it as "stable"? I know we've never done a milestone release in > the past, and I'm not sure what ASF guidance is on the topic, but if it > could be beneficial we could look into that. > > Cheers, > Kevin > > On Jan 11, 2023 at 12:22:43, Kevin Doran wrote: > > > I think this is a good, practical discussion. > > > > On the one hand, we can't put off 2.x any longer as we really need to > > updated the minimum required Java to 11. Updating main development to > > target 2.x feels like a good way drive progress on that along with the > > other 2.0 goals. > > > > On the other hand, the concerns are valid: moving all development to > > target 2.x puts the project at risk if we cannot release 2.0.0 on a > > reasonable timeline. The restricted scope of 2.0 helps, but this stated > > release goal feels risky to me: > > > > Implement Migration Tools for Upgrading Flows > > > > > >- Implement automated migration where possible to remap properties and > > features > > - Implement migration tools for manual conversion of XML Templates > > to JSON Flow Definitions > > - Create documentation for manual steps necessary where > > programmatic migration cannot be implemented > > - NiFi 2.0 should be capable of starting with ghosted components > > for removed Processors or Controller Services > > > > > > Removing deprecated components should be fairly straightforward and > quick, > > but automating and documenting migration is a large effort. > > > > On this po > > > > > > On Jan 10, 2023 at 09:32:31, Bryan Bende wrote: > > > >> The plan as I understand it is not to diverge and create separate > feature > >> development on the 1.x line, so I would expect all PRs to continue to be > >> submitted only to main. We would release 1.x as needed with major bug > >> fixes > >> or critical security updates, and these would be cherry-picked and/or > >> backported as necessary, mostly without the need for PRs, the same as we > >> would do if we were bringing fixes from main (1.20.0-SNAPSHOT) back to a > >> maintenance line like (1.19.x). For precedent, we followed this same > >> approach going from the 0.x line to 1.0.0 and there wasn't any major > >> issue. > >> > >> On Tue, Jan 10, 2023 at 7:07 AM Otto Fowler > >> wrote: > >> > >> It was also mentioned in another thread that we need to have agreement > on > >> > >> our explicit strategy and support for 1.x going forward, did that > happen? > >> > >> > >> From: Otto Fowler > >> > >> Reply: Otto Fowler > >> > >> Date: January 10, 2023 at 07:02:34 > >> > >> To: dev@nifi.apache.org > >> > >> Subject: Re: [discuss] NiFi 1.20 and NiFi 2.0 > >> > >> > >> There needs to be an update to the contributing guide as to how to > submit > >> > >> PRs to 1.x or 2.x etc. > >> > >> > >> From: Joe Witt > >> > >> Reply: dev@nifi.apache.org > >> > >> Date: January 9, 2023 at 15:53:16 > >> > >> To: dev@nifi.apache.org > >> > >> Subject: [discuss] NiFi 1.20 and NiFi 2.0 > >> > >> > >> Team, > >> > >> > >> As David mentioned in [1] following a successful NiFi 2.0 release goal > >> > >> planning - we are now going to start moving the 'main' line to be the > NiFi > >> > >> 2.0 line which will allow for the key work to take place. We will also > >> > >> move niFi 1.x to its appropriate support line. > >> > >> > >> It is also the case that we have nearly 100 JIRAs on NiFi 1.20 and we > have > >> > >> work in there including security items so it is time to make a release. > >> > >> The intent then is to initiate 1.20 and immediate after that change > 'main' > >> > >> to 2.0. > >> > >> > >> Going forward then all work on the 1.x line should be focused on > >> > >> maintaining existing features, dependencies, and helping 1.x users > migrate > >> > >> to the 2.x line. Otherwise, new feature work will happen on 'main' as it > >> > >> normally does and will come out in the NiFi 2.x release line. > >> > >> > >> Please flag key outstanding items as we narrow down the release > candidate > >> > >> for NiFi 1.20. > >> > >>
Re: [discuss] NiFi 1.20 and NiFi 2.0
Sorry, Joe, I was not clear, and to be honest the two thoughts are somewhat unrelated in my mind too :) I agree that good migration tooling is key. Otherwise, we risk users staying on 1.x or creating a schism of 1.x and 2.x users. Good migration tooling will take a while to develop and test, and the core contributors to that effort may not have sufficient variety of flows to evaluate when the migration tools are "done" for the majority of the community to have success upgrading to 2.x. A milestone release would allow us get more feedback on migration over a longer period than the vote window of an RC candidate. Perhaps we could continue to release from the 1.x line (including minor releases with some features) until we are ready to drop the "milestone" qualifier from 2.0.0, and only then put 1.x into maintenance-only status. It would be the same proposal to move main to target 2.0.0-M1, but relax restrictions for what can land on the 1.x branch and be open to a 1.21, 1.22, etc. if 2.0.0 work takes longer than anticipated. For example, maybe we would be open to landing new/backported processors on the 1.x branch, but not core framework features or API changes. This might not be necessary, but I think it is fair that saying "no new features on 1.x" and also "no new features in 2.0.0" puts the project in a rough place if 2.0.0 takes longer than a few months, so if we go that route, we need to commit to a quick release of 2.0.0 that most users can move to easily. Thanks, Kevin On Jan 11, 2023 at 12:32:46, Joe Witt wrote: > Kevin, > > Yeah we can do whatever we want as far as 'releases' of 2.0 that are prior > to us officially considering it 2.0/stable. > > That said - the migration tooling will be key to provide as we need to make > the bridge as solid and stable as possible to help someone move from 1.x to > 2.x. I dont know how related these two concepts (milestone releases and > 1.x to 2.x ease really are). > > Thanks > > On Wed, Jan 11, 2023 at 10:27 AM Kevin Doran wrote: > > [hit the wrong keyboard shortcut, here is the rest of my thoughts] > > > On this point from David: > > > We may need to have a longer release candidate period, or more incremental > > > fix releases > > > for the initial 2.0.0 release train, but I do not expect delaying a 2.0.0 > > > release for new features, as that is not part of the release goals. > > > > > > Would the community benefit from one or more milestone releases of 2.0, to > > allow for a wider group to run / live on the proposed 2.0 prior to > > releasing it as "stable"? I know we've never done a milestone release in > > the past, and I'm not sure what ASF guidance is on the topic, but if it > > could be beneficial we could look into that. > > > Cheers, > > Kevin > > > On Jan 11, 2023 at 12:22:43, Kevin Doran wrote: > > > > I think this is a good, practical discussion. > > > > > > On the one hand, we can't put off 2.x any longer as we really need to > > > updated the minimum required Java to 11. Updating main development to > > > target 2.x feels like a good way drive progress on that along with the > > > other 2.0 goals. > > > > > > On the other hand, the concerns are valid: moving all development to > > > target 2.x puts the project at risk if we cannot release 2.0.0 on a > > > reasonable timeline. The restricted scope of 2.0 helps, but this stated > > > release goal feels risky to me: > > > > > > Implement Migration Tools for Upgrading Flows > > > > > > > > >- Implement automated migration where possible to remap properties and > > > features > > > - Implement migration tools for manual conversion of XML Templates > > > to JSON Flow Definitions > > > - Create documentation for manual steps necessary where > > > programmatic migration cannot be implemented > > > - NiFi 2.0 should be capable of starting with ghosted components > > > for removed Processors or Controller Services > > > > > > > > > Removing deprecated components should be fairly straightforward and > > quick, > > > but automating and documenting migration is a large effort. > > > > > > On this po > > > > > > > > > On Jan 10, 2023 at 09:32:31, Bryan Bende wrote: > > > > > >> The plan as I understand it is not to diverge and create separate > > feature > > >> development on the 1.x line, so I would expect all PRs to continue to be > > >> submitted only to main. We would release 1.x as needed with major bug > > >> fixes > > >> or critical security updates, and these would be cherry-picked and/or > > >> backported as necessary, mostly without the need for PRs, the same as we > > >> would do if we were bringing fixes from main (1.20.0-SNAPSHOT) back to a > > >> maintenance line like (1.19.x). For precedent, we followed this same > > >> approach going from the 0.x line to 1.0.0 and there wasn't any major > > >> issue. > > >> > > >> On Tue, Jan 10, 2023 at 7:07 AM Otto Fowler > > >> wrote: > > >> > > >> It was also mentioned in another thread that we ne
Re: [discuss] NiFi 1.20 and NiFi 2.0
Super, is that written down anywhere we can point people to? From: Bryan Bende Reply: dev@nifi.apache.org Date: January 10, 2023 at 09:32:51 To: dev@nifi.apache.org Subject: Re: [discuss] NiFi 1.20 and NiFi 2.0 The plan as I understand it is not to diverge and create separate feature development on the 1.x line, so I would expect all PRs to continue to be submitted only to main. We would release 1.x as needed with major bug fixes or critical security updates, and these would be cherry-picked and/or backported as necessary, mostly without the need for PRs, the same as we would do if we were bringing fixes from main (1.20.0-SNAPSHOT) back to a maintenance line like (1.19.x). For precedent, we followed this same approach going from the 0.x line to 1.0.0 and there wasn't any major issue. On Tue, Jan 10, 2023 at 7:07 AM Otto Fowler wrote: > It was also mentioned in another thread that we need to have agreement on > our explicit strategy and support for 1.x going forward, did that happen? > > From: Otto Fowler > Reply: Otto Fowler > Date: January 10, 2023 at 07:02:34 > To: dev@nifi.apache.org > Subject: Re: [discuss] NiFi 1.20 and NiFi 2.0 > > There needs to be an update to the contributing guide as to how to submit > PRs to 1.x or 2.x etc. > > From: Joe Witt > Reply: dev@nifi.apache.org > Date: January 9, 2023 at 15:53:16 > To: dev@nifi.apache.org > Subject: [discuss] NiFi 1.20 and NiFi 2.0 > > Team, > > As David mentioned in [1] following a successful NiFi 2.0 release goal > planning - we are now going to start moving the 'main' line to be the NiFi > 2.0 line which will allow for the key work to take place. We will also > move niFi 1.x to its appropriate support line. > > It is also the case that we have nearly 100 JIRAs on NiFi 1.20 and we have > work in there including security items so it is time to make a release. > The intent then is to initiate 1.20 and immediate after that change 'main' > to 2.0. > > Going forward then all work on the 1.x line should be focused on > maintaining existing features, dependencies, and helping 1.x users migrate > to the 2.x line. Otherwise, new feature work will happen on 'main' as it > normally does and will come out in the NiFi 2.x release line. > > Please flag key outstanding items as we narrow down the release candidate > for NiFi 1.20. > > Thanks > Joe > > [1] https://lists.apache.org/thread/qo4vvdw46235y7vy2crcd6l4m11wl7jz >
Re: [discuss] NiFi 1.20 and NiFi 2.0
This is really insightful and spot on ... Kevin wrote: > Good migration tooling will take a while to develop and test, and the core > contributors to that effort may not have sufficient variety of flows to > evaluate when the migration tools are "done" for the majority of the > community to have success upgrading to 2.x. A milestone release would allow > us get more feedback on migration over a longer period than the vote window > of an RC candidate. It's exactly this case, that an early 2.0 release might not have had time to fully work its way through existing production deployments, that's concerning. The pace and voting of an "RC" is much too short to get any quality feedback from users in the field. I think it's really smart to consider the "Milestone" release approach here. We release 2.0.0-M1, 2.0.0-M2, ... waiting an adequate amount of time for feedback. We can put these milestones on a calendar, as needed, so that feedback is required some 'x' number of weeks/months after each milestone. And to this end, I'd personally rather see us keep the 'main' branch current with the 1.x line _until_ we're ready and are satisfied with the end goals of the 2.0 release objectives. When the milestone releases have been completed and there's a comfort level with the 2.x line, it's at the point we'd isolate the 1.x line into its own branch and switch main over to the 2.x line. This is an attractive way of: a) continuing business-as-usual with the 1.x line b) making headway on the 2.x release milestones c) giving adequate time for feedback against the 2.0 milestones coming from the field I don't mind the known-unknowns. But it's really the unknown-unknowns that are going to drive a delay in the 2.0 release. I think it's smart to be able to get some of the unknowns ironed out before we finalize the 2.0 release ceremony. The milestone approach really helps with that. /Adam On Wed, Jan 11, 2023 at 11:11 AM Kevin Doran wrote: > Sorry, Joe, I was not clear, and to be honest the two thoughts are somewhat > unrelated in my mind too :) > > I agree that good migration tooling is key. Otherwise, we risk users > staying on 1.x or creating a schism of 1.x and 2.x users. > > Good migration tooling will take a while to develop and test, and the core > contributors to that effort may not have sufficient variety of flows to > evaluate when the migration tools are "done" for the majority of the > community to have success upgrading to 2.x. A milestone release would allow > us get more feedback on migration over a longer period than the vote window > of an RC candidate. > > Perhaps we could continue to release from the 1.x line (including minor > releases with some features) until we are ready to drop the "milestone" > qualifier from 2.0.0, and only then put 1.x into maintenance-only status. > It would be the same proposal to move main to target 2.0.0-M1, but relax > restrictions for what can land on the 1.x branch and be open to a 1.21, > 1.22, etc. if 2.0.0 work takes longer than anticipated. For example, maybe > we would be open to landing new/backported processors on the 1.x branch, > but not core framework features or API changes. > > This might not be necessary, but I think it is fair that saying "no new > features on 1.x" and also "no new features in 2.0.0" puts the project in a > rough place if 2.0.0 takes longer than a few months, so if we go that > route, we need to commit to a quick release of 2.0.0 that most users can > move to easily. > > Thanks, > Kevin > > On Jan 11, 2023 at 12:32:46, Joe Witt wrote: > > > Kevin, > > > > Yeah we can do whatever we want as far as 'releases' of 2.0 that are > prior > > to us officially considering it 2.0/stable. > > > > That said - the migration tooling will be key to provide as we need to > make > > the bridge as solid and stable as possible to help someone move from 1.x > to > > 2.x. I dont know how related these two concepts (milestone releases and > > 1.x to 2.x ease really are). > > > > Thanks > > > > On Wed, Jan 11, 2023 at 10:27 AM Kevin Doran wrote: > > > > [hit the wrong keyboard shortcut, here is the rest of my thoughts] > > > > > > On this point from David: > > > > > > We may need to have a longer release candidate period, or more > incremental > > > > > fix releases > > > > > for the initial 2.0.0 release train, but I do not expect delaying a > 2.0.0 > > > > > release for new features, as that is not part of the release goals. > > > > > > > > > > > Would the community benefit from one or more milestone releases of 2.0, > to > > > > allow for a wider group to run / live on the proposed 2.0 prior to > > > > releasing it as "stable"? I know we've never done a milestone release in > > > > the past, and I'm not sure what ASF guidance is on the topic, but if it > > > > could be beneficial we could look into that. > > > > > > Cheers, > > > > Kevin > > > > > > On Jan 11, 2023 at 12:22:43, Kevin Doran wrote: > > > > > > > I think this is a good, practical discussion. >
PostHTTP Deprecation Concerns
Just wanted to note a concern on the deprecation (and presumed removal) of the PostHTTP processor in the upcoming 2.0 release. While yes, for traditional client interactions with an external HTTP service, utilizing InvokeHTTP for your POST operation is probably sensible. The concern is that there are a number of NiFi-to-NiFi transfers that leverage the "special sauce" that exists between PostHTTP and ListenHTTP. What special sauce? Namely, the extra negotiation that enables an automated serialization of NiFi flowfiles from one system to another. InvokeHTTP is just a "raw" HTTP client and doesn't share any special concern or support for NiFi-to-NiFi data transfer. Of course, if you remember the history, before there was any site-to-site functionality built into processor groups, the primary means of flowfile transport between NiFi systems was the PostHTTP / ListenHTTP combo. It was an easy way to facilitate transfer between two nifi systems. And from what I can tell, this "legacy" approach to NiFi data transfer is still being used heavily in certain operational contexts. Why? Because often it's the case that the _only_ traffic allowed between network boundaries is done via HTTPS. The site-to-site protocol provides its own ports and protocol operations that don't necessarily comply with such a network policy. And I believe there's still some lingering and/or demonstrated concern over the performance characteristics of the site-to-site protocol by dataflow managers. They have often reverted to using PostHTTP / ListenHTTP instead. While many of the other deprecated components seem logical, getting rid of this one just seems like change-for-the-sake-of-change. Is there any actual technical reason to deprecate and remove PostHTTP from the standard nar? Is it causing a burden to the product itself? Or was the decision just more like, "hey it's dumb not to use InvokeHTTP for all HTTP client operations" and maybe not realize the alternative use case that PostHTTP enables? Thanks for any feedback. /Adam
Problem with NIFI registry using ssl certificates
Dear,
After configuring the security parameters in the nifi registry, its operation
cannot be started since the logs indicate that the jetty web server could not
be started. Is there a solution for this issue or you must work with a special
version of the product so that it can be deployed with https.
Attached log output
==> nifi-registry-app_2023-01-11_12.0.log <==
at
org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
... 9 common frames omitted
Caused by: java.nio.channels.UnresolvedAddressException: null
at java.base/sun.nio.ch.Net.checkAddress(Net.java:131)
at
java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
... 10 common frames omitted
2023-01-11 12:56:44,477 INFO [Thread-0] org.apache.nifi.registry.NiFiRegistry
Initiating shutdown of Jetty web server...
2023-01-11 12:56:44,479 INFO [Thread-0]
o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@19e4653c{SSL,
(ssl, http/1.1)}{localhost :18443}
2023-01-11 12:56:44,479 INFO [Thread-0] org.eclipse.jetty.server.session node0
Stopped scavenging
Best regards,
--
Edison F. Naranjo E.
Seguridad Lógica
TELCONET LATAM
Cel: +593998608233
Quito-Ecuador
efnara...@telconet.ec
Toda la información contenida en este correo electrónico es confidencial y
podrá ser usada únicamente por los destinatarios. No imprimir a menos que sea
imprescindible.
Re: Problem with NIFI registry using ssl certificates
Hi Edison,
It sounds like your nifi-registry.properties file may have issues. Can you
share this section of configuration nifi.registry.web.https.host=?
nifi.registry.web.https.port=?
This guide should be able to help:
https://community.cloudera.com/t5/Community-Articles/Setting-Up-a-Secure-Apache-NiFi-Registry/ta-p/247753
There may be more exception information you can share with us that's
above/below the message you provided.
Nathan
On Wed, Jan 11, 2023, 6:21 PM EDISON FABRICIO NARANJO ESPIN <
efnara...@telconet.ec> wrote:
> Dear,
>
> After configuring the security parameters in the nifi registry, its
> operation cannot be started since the logs indicate that the jetty web
> server could not be started. Is there a solution for this issue or you must
> work with a special version of the product so that it can be deployed with
> https.
>
> Attached log output
>
> ==> nifi-registry-app_2023-01-11_12.0.log <==
> at
> org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
> ... 9 common frames omitted
> Caused by: java.nio.channels.UnresolvedAddressException: null
> at java.base/sun.nio.ch.Net.checkAddress(Net.java:131)
> at
> java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
> at
> java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
> ... 10 common frames omitted
> 2023-01-11 12:56:44,477 INFO [Thread-0]
> org.apache.nifi.registry.NiFiRegistry Initiating shutdown of Jetty web
> server...
> 2023-01-11 12:56:44,479 INFO [Thread-0]
> o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@19e4653c{SSL,
> (ssl, http/1.1)}{localhost :18443}
> 2023-01-11 12:56:44,479 INFO [Thread-0] org.eclipse.jetty.server.session
> node0 Stopped scavenging
>
>
> Best regards,
> --
> Edison F. Naranjo E.
> Seguridad Lógica
> TELCONET LATAM
> Cel: +593998608233
> Quito-Ecuador
> efnara...@telconet.ec
>
> Toda la información contenida en este correo electrónico es confidencial y
> podrá ser usada únicamente por los destinatarios. No imprimir a menos que
> sea imprescindible.
>
>
Re: [discuss] NiFi 1.20 and NiFi 2.0
Hello We dont have a release schedule and never really have. What we do have is 7+ years of track record which shows who and what we do. We have done a major release change before. We release every couple of months or so. This is very manageable and the stated goals are scoped as such. We have declared what our goals are for 2.0 and now it is time to simply make it happen. We will keep 1.x moving along in the meantime but one of these is going to get the ‘most’ energy as that is ultimately how all this works. Whatever is our ‘main’ line is where all PRs go by default and where people work by default. The burden of maintaining these lines is very real and will provide plenty of motivation to move 2.0 along rapidly. The true notion of business as usual is where PRs land and who does the work to review and merge them. The commentary around production usage worries suggests ideas or views that differ from the discussed, documented and now voted upon release goals. Someone starting on 2.0 should enjoy stability similar to a NiFi 1.20 release. Existing users will be provided tooling/automation/docs to move from 1.x to 2.x. Milestones may serve as a valuable tool for us. We can use them as we vet out migration tooling we put on the 1.x line. Do we need anything else for 1.20? Thanks On Wed, Jan 11, 2023 at 3:42 PM Adam Taft wrote: > This is really insightful and spot on ... > > Kevin wrote: > > Good migration tooling will take a while to develop and test, and the > core > > contributors to that effort may not have sufficient variety of flows to > > evaluate when the migration tools are "done" for the majority of the > > community to have success upgrading to 2.x. A milestone release would > allow > > us get more feedback on migration over a longer period than the vote > window > > of an RC candidate. > > It's exactly this case, that an early 2.0 release might not have had time > to fully work its way through existing production deployments, that's > concerning. The pace and voting of an "RC" is much too short to get any > quality feedback from users in the field. > > I think it's really smart to consider the "Milestone" release approach > here. We release 2.0.0-M1, 2.0.0-M2, ... waiting an adequate amount of time > for feedback. We can put these milestones on a calendar, as needed, so that > feedback is required some 'x' number of weeks/months after each milestone. > > And to this end, I'd personally rather see us keep the 'main' branch > current with the 1.x line _until_ we're ready and are satisfied with the > end goals of the 2.0 release objectives. When the milestone releases have > been completed and there's a comfort level with the 2.x line, it's at the > point we'd isolate the 1.x line into its own branch and switch main over to > the 2.x line. > > This is an attractive way of: > a) continuing business-as-usual with the 1.x line > b) making headway on the 2.x release milestones > c) giving adequate time for feedback against the 2.0 milestones coming from > the field > > I don't mind the known-unknowns. But it's really the unknown-unknowns that > are going to drive a delay in the 2.0 release. I think it's smart to be > able to get some of the unknowns ironed out before we finalize the 2.0 > release ceremony. The milestone approach really helps with that. > > /Adam > > On Wed, Jan 11, 2023 at 11:11 AM Kevin Doran wrote: > > > Sorry, Joe, I was not clear, and to be honest the two thoughts are > somewhat > > unrelated in my mind too :) > > > > I agree that good migration tooling is key. Otherwise, we risk users > > staying on 1.x or creating a schism of 1.x and 2.x users. > > > > Good migration tooling will take a while to develop and test, and the > core > > contributors to that effort may not have sufficient variety of flows to > > evaluate when the migration tools are "done" for the majority of the > > community to have success upgrading to 2.x. A milestone release would > allow > > us get more feedback on migration over a longer period than the vote > window > > of an RC candidate. > > > > Perhaps we could continue to release from the 1.x line (including minor > > releases with some features) until we are ready to drop the "milestone" > > qualifier from 2.0.0, and only then put 1.x into maintenance-only status. > > It would be the same proposal to move main to target 2.0.0-M1, but relax > > restrictions for what can land on the 1.x branch and be open to a 1.21, > > 1.22, etc. if 2.0.0 work takes longer than anticipated. For example, > maybe > > we would be open to landing new/backported processors on the 1.x branch, > > but not core framework features or API changes. > > > > This might not be necessary, but I think it is fair that saying "no new > > features on 1.x" and also "no new features in 2.0.0" puts the project in > a > > rough place if 2.0.0 takes longer than a few months, so if we go that > > route, we need to commit to a quick release of 2.0.0 that most users can > > move to easi
Re: [discuss] NiFi 1.20 and NiFi 2.0
I agree main needs to target 2.x, that is where most effort should be focused, and development there can be cherry picked / backported to a 1.x branch. The inverse approach creates too much of a moving target for 2.x. Joe, I do not see anything else needed for cutting 1.20. From: Joe Witt Sent: Wednesday, January 11, 2023 7:05:59 PM To: dev@nifi.apache.org Subject: Re: [discuss] NiFi 1.20 and NiFi 2.0 Hello We dont have a release schedule and never really have. What we do have is 7+ years of track record which shows who and what we do. We have done a major release change before. We release every couple of months or so. This is very manageable and the stated goals are scoped as such. We have declared what our goals are for 2.0 and now it is time to simply make it happen. We will keep 1.x moving along in the meantime but one of these is going to get the ‘most’ energy as that is ultimately how all this works. Whatever is our ‘main’ line is where all PRs go by default and where people work by default. The burden of maintaining these lines is very real and will provide plenty of motivation to move 2.0 along rapidly. The true notion of business as usual is where PRs land and who does the work to review and merge them. The commentary around production usage worries suggests ideas or views that differ from the discussed, documented and now voted upon release goals. Someone starting on 2.0 should enjoy stability similar to a NiFi 1.20 release. Existing users will be provided tooling/automation/docs to move from 1.x to 2.x. Milestones may serve as a valuable tool for us. We can use them as we vet out migration tooling we put on the 1.x line. Do we need anything else for 1.20? Thanks On Wed, Jan 11, 2023 at 3:42 PM Adam Taft wrote: > This is really insightful and spot on ... > > Kevin wrote: > > Good migration tooling will take a while to develop and test, and the > core > > contributors to that effort may not have sufficient variety of flows to > > evaluate when the migration tools are "done" for the majority of the > > community to have success upgrading to 2.x. A milestone release would > allow > > us get more feedback on migration over a longer period than the vote > window > > of an RC candidate. > > It's exactly this case, that an early 2.0 release might not have had time > to fully work its way through existing production deployments, that's > concerning. The pace and voting of an "RC" is much too short to get any > quality feedback from users in the field. > > I think it's really smart to consider the "Milestone" release approach > here. We release 2.0.0-M1, 2.0.0-M2, ... waiting an adequate amount of time > for feedback. We can put these milestones on a calendar, as needed, so that > feedback is required some 'x' number of weeks/months after each milestone. > > And to this end, I'd personally rather see us keep the 'main' branch > current with the 1.x line _until_ we're ready and are satisfied with the > end goals of the 2.0 release objectives. When the milestone releases have > been completed and there's a comfort level with the 2.x line, it's at the > point we'd isolate the 1.x line into its own branch and switch main over to > the 2.x line. > > This is an attractive way of: > a) continuing business-as-usual with the 1.x line > b) making headway on the 2.x release milestones > c) giving adequate time for feedback against the 2.0 milestones coming from > the field > > I don't mind the known-unknowns. But it's really the unknown-unknowns that > are going to drive a delay in the 2.0 release. I think it's smart to be > able to get some of the unknowns ironed out before we finalize the 2.0 > release ceremony. The milestone approach really helps with that. > > /Adam > > On Wed, Jan 11, 2023 at 11:11 AM Kevin Doran wrote: > > > Sorry, Joe, I was not clear, and to be honest the two thoughts are > somewhat > > unrelated in my mind too :) > > > > I agree that good migration tooling is key. Otherwise, we risk users > > staying on 1.x or creating a schism of 1.x and 2.x users. > > > > Good migration tooling will take a while to develop and test, and the > core > > contributors to that effort may not have sufficient variety of flows to > > evaluate when the migration tools are "done" for the majority of the > > community to have success upgrading to 2.x. A milestone release would > allow > > us get more feedback on migration over a longer period than the vote > window > > of an RC candidate. > > > > Perhaps we could continue to release from the 1.x line (including minor > > releases with some features) until we are ready to drop the "milestone" > > qualifier from 2.0.0, and only then put 1.x into maintenance-only status. > > It would be the same proposal to move main to target 2.0.0-M1, but relax > > restrictions for what can land on the 1.x branch and be open to a 1.21, > > 1.22, etc. if 2.0.0 work takes longer than anticipated. For example, > maybe > > we would
Re: PostHTTP Deprecation Concerns
Hi Adam, PostHTTP was marked deprecated 3 years ago (aka six technology lifetimes). The successive technologies to replace it's functionality are well documented and proven in production. The technical reason to remove it is that it is superfluous code that has a cost to maintain and zero benefit. Backwards compatibility is never guaranteed for components marked deprecated for such a long length of time in any software product let alone nifi specifically. Your organisation is free to continue using the version of nifi it is on today and not take any further action. It is unhelpful to suggest every other organisation should be held back in progress because yours refuses to take the necessary flow maintenance action. One of the impetus for a major version upgrade is to specifically jettison deprecated components. It's quite remarkable you're advocating against standard practice presumably for your own convenience. Site to site connectivity is conducted with either raw sockets or http (which is https on secured nifi) so I'm highly skeptical there is any performance deprecation in InvokeHTTP or S2S over PostHTTP, given the former can take advantage of http/2 and the latter not. It's easy to monitor nifi and prove through metrics in any case. Sadly in enterprise environments this is sometimes necessary to defeat the political layer around change management. You can run records-based processing over either current method and it is ridiculously fast. The bottleneck in my last engagement ended up being network hardware limitations, not nifi. Having contributed in this domain, I also recommend tossing CompressContent into the flow to minimise bandwidth. On modern hardware the decompression is minimal time and you can plug a *lot* more data through in less CPU and wall clock time. It's easy to bench with DuplicateFlowfile on your production flow and metrics analysis, just make sure your provenance db has sufficient space. Kind regards, On Thu, Jan 12, 2023, 10:09 Adam Taft wrote: > Just wanted to note a concern on the deprecation (and presumed removal) of > the PostHTTP processor in the upcoming 2.0 release. > > While yes, for traditional client interactions with an external HTTP > service, utilizing InvokeHTTP for your POST operation is probably sensible. > The concern is that there are a number of NiFi-to-NiFi transfers that > leverage the "special sauce" that exists between PostHTTP and ListenHTTP. > > What special sauce? Namely, the extra negotiation that enables an automated > serialization of NiFi flowfiles from one system to another. InvokeHTTP is > just a "raw" HTTP client and doesn't share any special concern or support > for NiFi-to-NiFi data transfer. > > Of course, if you remember the history, before there was any site-to-site > functionality built into processor groups, the primary means of flowfile > transport between NiFi systems was the PostHTTP / ListenHTTP combo. It was > an easy way to facilitate transfer between two nifi systems. > > And from what I can tell, this "legacy" approach to NiFi data transfer is > still being used heavily in certain operational contexts. Why? Because > often it's the case that the _only_ traffic allowed between network > boundaries is done via HTTPS. The site-to-site protocol provides its own > ports and protocol operations that don't necessarily comply with such a > network policy. And I believe there's still some lingering and/or > demonstrated concern over the performance characteristics of the > site-to-site protocol by dataflow managers. They have often reverted to > using PostHTTP / ListenHTTP instead. > > While many of the other deprecated components seem logical, getting rid of > this one just seems like change-for-the-sake-of-change. > > Is there any actual technical reason to deprecate and remove PostHTTP from > the standard nar? Is it causing a burden to the product itself? Or was the > decision just more like, "hey it's dumb not to use InvokeHTTP for all HTTP > client operations" and maybe not realize the alternative use case that > PostHTTP enables? > > Thanks for any feedback. > > /Adam >
Re: PostHTTP Deprecation Concerns
Adam, This question has come up in some Jira issues over the years, but the majority of community interest and effort continues to focus on InvokeHTTP. There are several good reasons for removing PostHTTP, and recent versions of InvokeHTTP have included several important improvements to the InvokeHTTP-ListenHTTP flow. In addition to being deprecated for years, there are several technical reasons for removing in from the project in its current condition. 1. PostHTTP uses Apache HttpComponents HttpClient 4.5, which has been in maintenance mode for the last several years with the release of version 5 2. PostHTTP no longer has active test cases in Apache NiFi due its deprecation 3. PostHTTP includes an internal retry strategy that locks processing tasks with the use of Thread.sleep() As you noted, the Send as FlowFile feature of PostHTTP implements some specialized behavior for communicating with ListenHTTP. This is a convenient feature for sending both FlowFile content and attributes to another NiFi server running ListenHTTP, but it depends on specialized packaging. This same behavior can be achieved using the MergeContent Processor, and then sending the content through InvokeHTTP. Similar results can be achieved through sending FlowFile attributes as HTTP headers using InvokeHTTP. The custom processing in PostHTTP also performs Gzip compression during transmission. NiFi 1.17.0 introduced several improvements to InvokeHTTP and ListenHTTP that provide more generalized performance improvements. InvokeHTTP has a Request Content-Type property that can be set to GZIP, which implements a standards-based approach to compressing content during transmission. ListenHTTP automatically reads the Content-Type header and handles decompression as needed. NiFi 1.17.0 also added supporting for HTTP/2 in ListenHTTP, which uses binary formatting for HTTP headers, and supports multiplexing requests, as opposed to connection pooling strategies necessary for HTTP/1.1. InvokeHTTP and the OkHttp library have supported HTTP/2 in the default configuration for a number of versions. These improvements address several of the theoretical gaps between PostHTTP and InvokeHTTP sending to ListenHTTP. Nothing in the scope of NiFi 2.0 includes directly breaking support for older processors, so running older components is a last resort option. Regards, David Handermann On Wed, Jan 11, 2023 at 5:09 PM Adam Taft wrote: > Just wanted to note a concern on the deprecation (and presumed removal) of > the PostHTTP processor in the upcoming 2.0 release. > > While yes, for traditional client interactions with an external HTTP > service, utilizing InvokeHTTP for your POST operation is probably sensible. > The concern is that there are a number of NiFi-to-NiFi transfers that > leverage the "special sauce" that exists between PostHTTP and ListenHTTP. > > What special sauce? Namely, the extra negotiation that enables an automated > serialization of NiFi flowfiles from one system to another. InvokeHTTP is > just a "raw" HTTP client and doesn't share any special concern or support > for NiFi-to-NiFi data transfer. > > Of course, if you remember the history, before there was any site-to-site > functionality built into processor groups, the primary means of flowfile > transport between NiFi systems was the PostHTTP / ListenHTTP combo. It was > an easy way to facilitate transfer between two nifi systems. > > And from what I can tell, this "legacy" approach to NiFi data transfer is > still being used heavily in certain operational contexts. Why? Because > often it's the case that the _only_ traffic allowed between network > boundaries is done via HTTPS. The site-to-site protocol provides its own > ports and protocol operations that don't necessarily comply with such a > network policy. And I believe there's still some lingering and/or > demonstrated concern over the performance characteristics of the > site-to-site protocol by dataflow managers. They have often reverted to > using PostHTTP / ListenHTTP instead. > > While many of the other deprecated components seem logical, getting rid of > this one just seems like change-for-the-sake-of-change. > > Is there any actual technical reason to deprecate and remove PostHTTP from > the standard nar? Is it causing a burden to the product itself? Or was the > decision just more like, "hey it's dumb not to use InvokeHTTP for all HTTP > client operations" and maybe not realize the alternative use case that > PostHTTP enables? > > Thanks for any feedback. > > /Adam >
Re: PostHTTP Deprecation Concerns
Hi Mathew, > It's quite remarkable you're advocating against standard practice presumably > for your own convenience. Wow, absolutely not stated nor implied in my message. And even borderline offensive. What I asked was simply, why remove it, if it's not hurting anything. I agree with your statement that there is a (very small) cost for maintaining the component in the source tree. But PostHTTP is not in the same scope as compared to a component that has a dependency on an abandoned, insecure, or completely out of standards library (for example). PostHTTP has a reasonable use case (as I described) that is not directly matched with other processors. The two-phase commit protocol sitting between PostHTTP and ListenHTTP has demonstrated to bear good fruit over many hardened years of use. I think it's a reasonable reply to my question to just simply suggest that the interaction between PostHTTP and ListenHTTP is just not supported by NiFi going forward. But please don't tell me my question/concern is "out of convenience." There is lacking documentation as to the rationale behind the deprecation of PostHTTP. I might be missing it, can you please send me the link to the rationale? That's what this thread is trying to address. It sounds like, from your answer, that the rationale is to reduce code footprint, which isn't the strongest argument for its removal given its established historical use. Seems like we'd want more than just reduced footprint for such a heavily used processor, no? /Adam On Wed, Jan 11, 2023 at 7:53 PM Matthew Hawkins wrote: > Hi Adam, > > PostHTTP was marked deprecated 3 years ago (aka six technology lifetimes). > The successive technologies to replace it's functionality are well > documented and proven in production. The technical reason to remove it is > that it is superfluous code that has a cost to maintain and zero benefit. > Backwards compatibility is never guaranteed for components marked > deprecated for such a long length of time in any software product let alone > nifi specifically. > > Your organisation is free to continue using the version of nifi it is on > today and not take any further action. It is unhelpful to suggest every > other organisation should be held back in progress because yours refuses to > take the necessary flow maintenance action. One of the impetus for a major > version upgrade is to specifically jettison deprecated components. It's > quite remarkable you're advocating against standard practice presumably for > your own convenience. > > Site to site connectivity is conducted with either raw sockets or http > (which is https on secured nifi) so I'm highly skeptical there is any > performance deprecation in InvokeHTTP or S2S over PostHTTP, given the > former can take advantage of http/2 and the latter not. It's easy to > monitor nifi and prove through metrics in any case. Sadly in enterprise > environments this is sometimes necessary to defeat the political layer > around change management. > > You can run records-based processing over either current method and it is > ridiculously fast. The bottleneck in my last engagement ended up being > network hardware limitations, not nifi. Having contributed in this domain, > I also recommend tossing CompressContent into the flow to minimise > bandwidth. On modern hardware the decompression is minimal time and you can > plug a *lot* more data through in less CPU and wall clock time. It's easy > to bench with DuplicateFlowfile on your production flow and metrics > analysis, just make sure your provenance db has sufficient space. > > Kind regards, > > On Thu, Jan 12, 2023, 10:09 Adam Taft wrote: > > > Just wanted to note a concern on the deprecation (and presumed removal) > of > > the PostHTTP processor in the upcoming 2.0 release. > > > > While yes, for traditional client interactions with an external HTTP > > service, utilizing InvokeHTTP for your POST operation is probably > sensible. > > The concern is that there are a number of NiFi-to-NiFi transfers that > > leverage the "special sauce" that exists between PostHTTP and ListenHTTP. > > > > What special sauce? Namely, the extra negotiation that enables an > automated > > serialization of NiFi flowfiles from one system to another. InvokeHTTP is > > just a "raw" HTTP client and doesn't share any special concern or support > > for NiFi-to-NiFi data transfer. > > > > Of course, if you remember the history, before there was any site-to-site > > functionality built into processor groups, the primary means of flowfile > > transport between NiFi systems was the PostHTTP / ListenHTTP combo. It > was > > an easy way to facilitate transfer between two nifi systems. > > > > And from what I can tell, this "legacy" approach to NiFi data transfer is > > still being used heavily in certain operational contexts. Why? Because > > often it's the case that the _only_ traffic allowed between network > > boundaries is done via HTTPS. The site-to-site protocol provides its own > >
Re: PostHTTP Deprecation Concerns
David,
Thank you for the reasonable response to my questions. Much appreciated.
I'm not a huge fan of the MergeContent -> InvokeHTTP -> {} -> ListenHTTP ->
UnpackContent approach to provide the same functionality. But I do
acknowledge that's the most direct replacement option without PostHTTP.
It's adding extract processors to the chain for something that is
effectively a transport issue. NiFi-to-Nifi using PostHTTP was a simple
transport-oriented solution, and packing the data with MergeContent first
isn't quite the same level of fidelity. You also miss the two-phase commit
built into those extra bits. MergeContent is often a bit of a beast
in-and-of-itself too.
Flowfile attributes conveyed as HTTP headers definitely don't work for
complex attribute values. But yes, I know that the functionality exists
(having some history with that processor myself).
Thanks again for the response.
/Adam
On Wed, Jan 11, 2023 at 9:27 PM Adam Taft wrote:
> Hi Mathew,
>
> > It's quite remarkable you're advocating against standard practice
> presumably
> > for your own convenience.
>
> Wow, absolutely not stated nor implied in my message. And even borderline
> offensive.
>
> What I asked was simply, why remove it, if it's not hurting anything. I
> agree with your statement that there is a (very small) cost for maintaining
> the component in the source tree. But PostHTTP is not in the same scope as
> compared to a component that has a dependency on an abandoned, insecure, or
> completely out of standards library (for example).
>
> PostHTTP has a reasonable use case (as I described) that is not directly
> matched with other processors. The two-phase commit protocol sitting
> between PostHTTP and ListenHTTP has demonstrated to bear good fruit over
> many hardened years of use. I think it's a reasonable reply to my question
> to just simply suggest that the interaction between PostHTTP and ListenHTTP
> is just not supported by NiFi going forward. But please don't tell me my
> question/concern is "out of convenience."
>
> There is lacking documentation as to the rationale behind the deprecation
> of PostHTTP. I might be missing it, can you please send me the link to the
> rationale? That's what this thread is trying to address. It sounds like,
> from your answer, that the rationale is to reduce code footprint, which
> isn't the strongest argument for its removal given its established
> historical use. Seems like we'd want more than just reduced footprint for
> such a heavily used processor, no?
>
> /Adam
>
>
> On Wed, Jan 11, 2023 at 7:53 PM Matthew Hawkins
> wrote:
>
>> Hi Adam,
>>
>> PostHTTP was marked deprecated 3 years ago (aka six technology lifetimes).
>> The successive technologies to replace it's functionality are well
>> documented and proven in production. The technical reason to remove it is
>> that it is superfluous code that has a cost to maintain and zero benefit.
>> Backwards compatibility is never guaranteed for components marked
>> deprecated for such a long length of time in any software product let
>> alone
>> nifi specifically.
>>
>> Your organisation is free to continue using the version of nifi it is on
>> today and not take any further action. It is unhelpful to suggest every
>> other organisation should be held back in progress because yours refuses
>> to
>> take the necessary flow maintenance action. One of the impetus for a major
>> version upgrade is to specifically jettison deprecated components. It's
>> quite remarkable you're advocating against standard practice presumably
>> for
>> your own convenience.
>>
>> Site to site connectivity is conducted with either raw sockets or http
>> (which is https on secured nifi) so I'm highly skeptical there is any
>> performance deprecation in InvokeHTTP or S2S over PostHTTP, given the
>> former can take advantage of http/2 and the latter not. It's easy to
>> monitor nifi and prove through metrics in any case. Sadly in enterprise
>> environments this is sometimes necessary to defeat the political layer
>> around change management.
>>
>> You can run records-based processing over either current method and it is
>> ridiculously fast. The bottleneck in my last engagement ended up being
>> network hardware limitations, not nifi. Having contributed in this domain,
>> I also recommend tossing CompressContent into the flow to minimise
>> bandwidth. On modern hardware the decompression is minimal time and you
>> can
>> plug a *lot* more data through in less CPU and wall clock time. It's easy
>> to bench with DuplicateFlowfile on your production flow and metrics
>> analysis, just make sure your provenance db has sufficient space.
>>
>> Kind regards,
>>
>> On Thu, Jan 12, 2023, 10:09 Adam Taft wrote:
>>
>> > Just wanted to note a concern on the deprecation (and presumed removal)
>> of
>> > the PostHTTP processor in the upcoming 2.0 release.
>> >
>> > While yes, for traditional client interactions with an external HTTP
>> > service, utilizing InvokeHTTP for your
