Disabling flows - nifi registry
Hey guys, So, once I fully develop and parameterize my nifi dataflow, let’s say in dev environment, I enable the version control, import the flow in higher environment and turn on the dataflow. In most of the cases both the flows in lower and higher environments will be running. Let says dev nifi connects to dev gcp pubsub and prod nifi connects to prod gcp pubsub. However, in some cases, we do want to stop and disable the flow in lower env. When I do that the registry is identifying that local changes are made to the flow which is nothing but all the components are disabled. I don’t want to keep the processors in stopped state on the canvas(registry do not identify for stopping) but want to disable them. Any workaround for registry not to identify local changes when flow is disabled? Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Re: Disabling flows - nifi registry
Hi Deepak, So far, we have been honoring the following policy for what constitutes a change in version control: - stopped/started does not count as a "local change" - enabled/disabled does count as a change, and that state is captured in the flow snapshot json version saved to registry. One reason for this is that some users want to setup CI/CD to deploy from Registry and automatically start a flow in the target NiFi. If there are components they don't want to start, the disabled state gives them a way to capture that configuration in the flow. Is there a reason you prefer disable over stop in your lower environment? I leave flows stopped in dev environments all the time and have never run into an issue, but of course, everyone's workflow and use case is slightly different, so I'm interested in hearing your perspective on this to see if we need to consider something more flexible. Cheers, Kevin On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" < c-deepakreddy.chirth...@charter.com> wrote: > Hey guys, > > So, once I fully develop and parameterize my nifi dataflow, let’s say in > dev environment, I enable the version control, import the flow in higher > environment and turn on the dataflow. In most of the cases both the flows > in lower and higher environments will be running. Let says dev nifi > connects to dev gcp pubsub and prod nifi connects to prod gcp pubsub. > However, in some cases, we do want to stop and disable the flow in lower > env. When I do that the registry is identifying that local changes are made > to the flow which is nothing but all the components are disabled. I don’t > want to keep the processors in stopped state on the canvas(registry do not > identify for stopping) but want to disable them. Any workaround for > registry not to identify local changes when flow is disabled? > > > > Thanks > > Deepak > > > > *[image: image005]* > > *Deepak Reddy* | Data Engineer > IT Centers of Excellence > 13736 Riverport Dr., Maryland Heights, MO 63043 > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. >
RE: [EXTERNAL] Re: Disabling flows - nifi registry
Hi Kevin, Thank you for your quick response. The only reason why I prefer processors to be disabled is to improve the NiFi UI performance by simply disabling stopped processors which I am not using. https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550 I had issues with Nifi UI performance in the past where the refresh wheel keeps on spinning like for 10 seconds and disabling stopped processors really helped me. Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Kevin Doran Sent: Friday, December 9, 2022 9:33 AM To: users@nifi.apache.org Subject: [EXTERNAL] Re: Disabling flows - nifi registry CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi Deepak, So far, we have been honoring the following policy for what constitutes a change in version control: * stopped/started does not count as a "local change" * enabled/disabled does count as a change, and that state is captured in the flow snapshot json version saved to registry. One reason for this is that some users want to setup CI/CD to deploy from Registry and automatically start a flow in the target NiFi. If there are components they don't want to start, the disabled state gives them a way to capture that configuration in the flow. Is there a reason you prefer disable over stop in your lower environment? I leave flows stopped in dev environments all the time and have never run into an issue, but of course, everyone's workflow and use case is slightly different, so I'm interested in hearing your perspective on this to see if we need to consider something more flexible. Cheers, Kevin On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" mailto:c-deepakreddy.chirth...@charter.com>> wrote: Hey guys, So, once I fully develop and parameterize my nifi dataflow, let’s say in dev environment, I enable the version control, import the flow in higher environment and turn on the dataflow. In most of the cases both the flows in lower and higher environments will be running. Let says dev nifi connects to dev gcp pubsub and prod nifi connects to prod gcp pubsub. However, in some cases, we do want to stop and disable the flow in lower env. When I do that the registry is identifying that local changes are made to the flow which is nothing but all the components are disabled. I don’t want to keep the processors in stopped state on the canvas(registry do not identify for stopping) but want to disable them. Any workaround for registry not to identify local changes when flow is disabled? Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
RE: [EXTERNAL] Re: Disabling flows - nifi registry
Thanks for sharing Deepak. That's interesting. I think we may look into a way of disabling components that doesn't impact version control, or else fix performance issues such that disabling is not necessary. Just out of curiosity, what version of NiFi are you on? The reason I ask is that I know there have been some UI performance improvements made in recent versions of NiFi, so I'm wondering if some of this might already have been addressed... Thanks, Kevin On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" < c-deepakreddy.chirth...@charter.com> wrote: > Hi Kevin, > > Thank you for your quick response. The only reason why I prefer processors > to be disabled is to improve the NiFi UI performance by simply disabling > stopped processors which I am not using. > > > https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550 > > I had issues with Nifi UI performance in the past where the refresh wheel > keeps on spinning like for 10 seconds and disabling stopped processors > really helped me. > > Thanks > Deepak > > > > > > *[image: image005]* > > *Deepak Reddy* | Data Engineer > IT Centers of Excellence > 13736 Riverport Dr., Maryland Heights, MO 63043 > > > > *From:* Kevin Doran > *Sent:* Friday, December 9, 2022 9:33 AM > *To:* users@nifi.apache.org > *Subject:* [EXTERNAL] Re: Disabling flows - nifi registry > > > > *CAUTION:* The e-mail below is from an external source. Please exercise > caution before opening attachments, clicking links, or following guidance. > > Hi Deepak, > > > > So far, we have been honoring the following policy for what constitutes a > change in version control: > > > >- stopped/started does not count as a "local change" >- enabled/disabled does count as a change, and that state is captured >in the flow snapshot json version saved to registry. > > > > One reason for this is that some users want to setup CI/CD to deploy from > Registry and automatically start a flow in the target NiFi. If there are > components they don't want to start, the disabled state gives them a way to > capture that configuration in the flow. > > > > Is there a reason you prefer disable over stop in your lower environment? > I leave flows stopped in dev environments all the time and have never run > into an issue, but of course, everyone's workflow and use case is slightly > different, so I'm interested in hearing your perspective on this to see if > we need to consider something more flexible. > > > > Cheers, > > Kevin > > > > On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" < > c-deepakreddy.chirth...@charter.com> wrote: > > Hey guys, > > So, once I fully develop and parameterize my nifi dataflow, let’s say in > dev environment, I enable the version control, import the flow in higher > environment and turn on the dataflow. In most of the cases both the flows > in lower and higher environments will be running. Let says dev nifi > connects to dev gcp pubsub and prod nifi connects to prod gcp pubsub. > However, in some cases, we do want to stop and disable the flow in lower > env. When I do that the registry is identifying that local changes are made > to the flow which is nothing but all the components are disabled. I don’t > want to keep the processors in stopped state on the canvas(registry do not > identify for stopping) but want to disable them. Any workaround for > registry not to identify local changes when flow is disabled? > > > > Thanks > > Deepak > > > > *[image: image005]* > > *Deepak Reddy* | Data Engineer > IT Centers of Excellence > 13736 Riverport Dr., Maryland Heights, MO 63043 > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. >
RE: [EXTERNAL] Re: Disabling flows - nifi registry
The version of Nifi where I had UI performance issues was 1.11.4 The current version of our nifi instance is 1.15.3 One other disadvantage of having Process Groups with Stopped components is that when a dataflow manager or a developer inadvertently right click and select “start” on the root canvas page, that will start all the components in the child process groups. Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Kevin Doran Sent: Tuesday, December 13, 2022 10:26 AM To: users@nifi.apache.org Subject: RE: [EXTERNAL] Re: Disabling flows - nifi registry CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Thanks for sharing Deepak. That's interesting. I think we may look into a way of disabling components that doesn't impact version control, or else fix performance issues such that disabling is not necessary. Just out of curiosity, what version of NiFi are you on? The reason I ask is that I know there have been some UI performance improvements made in recent versions of NiFi, so I'm wondering if some of this might already have been addressed... Thanks, Kevin On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" mailto:c-deepakreddy.chirth...@charter.com>> wrote: Hi Kevin, Thank you for your quick response. The only reason why I prefer processors to be disabled is to improve the NiFi UI performance by simply disabling stopped processors which I am not using. https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550 I had issues with Nifi UI performance in the past where the refresh wheel keeps on spinning like for 10 seconds and disabling stopped processors really helped me. Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Kevin Doran mailto:kdo...@apache.org>> Sent: Friday, December 9, 2022 9:33 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: [EXTERNAL] Re: Disabling flows - nifi registry CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi Deepak, So far, we have been honoring the following policy for what constitutes a change in version control: 1. stopped/started does not count as a "local change" 2. enabled/disabled does count as a change, and that state is captured in the flow snapshot json version saved to registry. One reason for this is that some users want to setup CI/CD to deploy from Registry and automatically start a flow in the target NiFi. If there are components they don't want to start, the disabled state gives them a way to capture that configuration in the flow. Is there a reason you prefer disable over stop in your lower environment? I leave flows stopped in dev environments all the time and have never run into an issue, but of course, everyone's workflow and use case is slightly different, so I'm interested in hearing your perspective on this to see if we need to consider something more flexible. Cheers, Kevin On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" mailto:c-deepakreddy.chirth...@charter.com>> wrote: Hey guys, So, once I fully develop and parameterize my nifi dataflow, let’s say in dev environment, I enable the version control, import the flow in higher environment and turn on the dataflow. In most of the cases both the flows in lower and higher environments will be running. Let says dev nifi connects to dev gcp pubsub and prod nifi connects to prod gcp pubsub. However, in some cases, we do want to stop and disable the flow in lower env. When I do that the registry is identifying that local changes are made to the flow which is nothing but all the components are disabled. I don’t want to keep the processors in stopped state on the canvas(registry do not identify for stopping) but want to disable them. Any workaround for registry not to identify local changes when flow is disabled? Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. The contents of this e-mail
RE: [EXTERNAL] Re: Disabling flows - nifi registry
Thanks, Deepak. I'll have to think about this one. I'm not sure if anyone else has any thoughts on this workflow / use case? On Dec 13, 2022 at 14:52:31, "Chirthani, Deepak Reddy" < c-deepakreddy.chirth...@charter.com> wrote: > The version of Nifi where I had UI performance issues was 1.11.4 > The current version of our nifi instance is 1.15.3 > > > One other disadvantage of having Process Groups with Stopped components is > that when a dataflow manager or a developer inadvertently right click and > select “start” on the root canvas page, that will start all the components > in the child process groups. > > Thanks > > Deepak > > *[image: image005]* > > *Deepak Reddy* | Data Engineer > IT Centers of Excellence > 13736 Riverport Dr., Maryland Heights, MO 63043 > > > > *From:* Kevin Doran > *Sent:* Tuesday, December 13, 2022 10:26 AM > *To:* users@nifi.apache.org > *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry > > > > *CAUTION:* The e-mail below is from an external source. Please exercise > caution before opening attachments, clicking links, or following guidance. > > Thanks for sharing Deepak. That's interesting. I think we may look into a > way of disabling components that doesn't impact version control, or else > fix performance issues such that disabling is not necessary. > > > > Just out of curiosity, what version of NiFi are you on? The reason I ask > is that I know there have been some UI performance improvements made in > recent versions of NiFi, so I'm wondering if some of this might already > have been addressed... > > > > Thanks, > > Kevin > > > > On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" < > c-deepakreddy.chirth...@charter.com> wrote: > > Hi Kevin, > > Thank you for your quick response. The only reason why I prefer processors > to be disabled is to improve the NiFi UI performance by simply disabling > stopped processors which I am not using. > > > https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550 > > I had issues with Nifi UI performance in the past where the refresh wheel > keeps on spinning like for 10 seconds and disabling stopped processors > really helped me. > > Thanks > Deepak > > > > > > *[image: image005]* > > *Deepak Reddy* | Data Engineer > IT Centers of Excellence > 13736 Riverport Dr., Maryland Heights, MO 63043 > > > > *From:* Kevin Doran > *Sent:* Friday, December 9, 2022 9:33 AM > *To:* users@nifi.apache.org > *Subject:* [EXTERNAL] Re: Disabling flows - nifi registry > > > > *CAUTION:* The e-mail below is from an external source. Please exercise > caution before opening attachments, clicking links, or following guidance. > > Hi Deepak, > > > > So far, we have been honoring the following policy for what constitutes a > change in version control: > > > >1. stopped/started does not count as a "local change" >2. enabled/disabled does count as a change, and that state is captured >in the flow snapshot json version saved to registry. > > > > One reason for this is that some users want to setup CI/CD to deploy from > Registry and automatically start a flow in the target NiFi. If there are > components they don't want to start, the disabled state gives them a way to > capture that configuration in the flow. > > > > Is there a reason you prefer disable over stop in your lower environment? > I leave flows stopped in dev environments all the time and have never run > into an issue, but of course, everyone's workflow and use case is slightly > different, so I'm interested in hearing your perspective on this to see if > we need to consider something more flexible. > > > > Cheers, > > Kevin > > > > On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" < > c-deepakreddy.chirth...@charter.com> wrote: > > Hey guys, > > So, once I fully develop and parameterize my nifi dataflow, let’s say in > dev environment, I enable the version control, import the flow in higher > environment and turn on the dataflow. In most of the cases both the flows > in lower and higher environments will be running. Let says dev nifi > connects to dev gcp pubsub and prod nifi connects to prod gcp pubsub. > However, in some cases, we do want to stop and disable the flow in lower > env. When I do that the registry is identifying that local changes are made > to the flow which is nothing but all the components are disabled. I don’t > want to keep the processors in stopped state on the canvas(regi
RE: [EXTERNAL] Re: Disabling flows - nifi registry
Personally I am against it. I feel that it is more important to protect production than development, and the operators running production will have less time and possibly less skill to look in depth at what should and should not be running, than the developers who are running development. My use case is that I want my operators to be able to press the run button without worrying about which processors or ports might need to be stopped. They should be able to observe the top level groups and see there are no stopped processors. As a developer, if I think a processor should not be run in production, then I will disable it before I commit it to the registry. These will include test injectors and fix/feedback loops that would be used often in development, but may also have a role in production during troubleshooting. The trigger to version control when the enable/disable state of a processor changes is an extra mechanism to warn if a processor has been used in production and has been accidentally left enabled. I prefer to review all changes in production before committing so that I am not accidentally committing untested “workarounds” that got me out of trouble this morning but would not like to persist in the code base. Steve Hindmarch From: Kevin Doran Sent: 15 December 2022 17:00 To: users@nifi.apache.org Subject: RE: [EXTERNAL] Re: Disabling flows - nifi registry Thanks, Deepak. I'll have to think about this one. I'm not sure if anyone else has any thoughts on this workflow / use case? On Dec 13, 2022 at 14:52:31, "Chirthani, Deepak Reddy" mailto:c-deepakreddy.chirth...@charter.com>> wrote: The version of Nifi where I had UI performance issues was 1.11.4 The current version of our nifi instance is 1.15.3 One other disadvantage of having Process Groups with Stopped components is that when a dataflow manager or a developer inadvertently right click and select “start” on the root canvas page, that will start all the components in the child process groups. Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Kevin Doran mailto:kdo...@apache.org>> Sent: Tuesday, December 13, 2022 10:26 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: RE: [EXTERNAL] Re: Disabling flows - nifi registry CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Thanks for sharing Deepak. That's interesting. I think we may look into a way of disabling components that doesn't impact version control, or else fix performance issues such that disabling is not necessary. Just out of curiosity, what version of NiFi are you on? The reason I ask is that I know there have been some UI performance improvements made in recent versions of NiFi, so I'm wondering if some of this might already have been addressed... Thanks, Kevin On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" mailto:c-deepakreddy.chirth...@charter.com>> wrote: Hi Kevin, Thank you for your quick response. The only reason why I prefer processors to be disabled is to improve the NiFi UI performance by simply disabling stopped processors which I am not using. https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.cloudera.com%2Ft5%2FSupport-Questions%2FNifi-UI-Working-very-Slow-How-to-increase-performance-of-a%2Ftd-p%2F214550&data=05%7C01%7Cstephen.hindmarch%40bt.com%7C86e782f0c8b747d242e208dadebddd53%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C638067204299548928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZVoWjub6gDAju1xyliaJXCZ%2BVcc5LF5NZn6%2FidKKe7s%3D&reserved=0> I had issues with Nifi UI performance in the past where the refresh wheel keeps on spinning like for 10 seconds and disabling stopped processors really helped me. Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Kevin Doran mailto:kdo...@apache.org>> Sent: Friday, December 9, 2022 9:33 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: [EXTERNAL] Re: Disabling flows - nifi registry CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi Deepak, So far, we have been honoring the following policy for what constitutes a change in version control: 1. stopped/started does not count as a "local change" 2. enabled/disabled does count as a change, and that state is captured in the flow snapshot json version saved to registry. One reason for this is that some u
Re: [EXTERNAL] Re: Disabling flows - nifi registry
Yea, the other side of this is when someone has disabled a portion of their flow in a lower environment and committed to registry, and then imports to their higher environment and expects that part to remain disabled. Otherwise that change has to be redone in higher environment. On Thu, Dec 15, 2022 at 12:55 PM stephen.hindmarch.bt.com via users < users@nifi.apache.org> wrote: > Personally I am against it. I feel that it is more important to protect > production than development, and the operators running production will have > less time and possibly less skill to look in depth at what should and > should not be running, than the developers who are running development. > > > > My use case is that I want my operators to be able to press the run button > without worrying about which processors or ports might need to be stopped. > They should be able to observe the top level groups and see there are no > stopped processors. > > > > As a developer, if I think a processor should not be run in production, > then I will disable it before I commit it to the registry. These will > include test injectors and fix/feedback loops that would be used often in > development, but may also have a role in production during troubleshooting. > The trigger to version control when the enable/disable state of a processor > changes is an extra mechanism to warn if a processor has been used in > production and has been accidentally left enabled. I prefer to review all > changes in production before committing so that I am not accidentally > committing untested “workarounds” that got me out of trouble this morning > but would not like to persist in the code base. > > > > *Steve Hindmarch* > > > > *From:* Kevin Doran > *Sent:* 15 December 2022 17:00 > *To:* users@nifi.apache.org > *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry > > > > Thanks, Deepak. I'll have to think about this one. I'm not sure if anyone > else has any thoughts on this workflow / use case? > > > > On Dec 13, 2022 at 14:52:31, "Chirthani, Deepak Reddy" < > c-deepakreddy.chirth...@charter.com> wrote: > > The version of Nifi where I had UI performance issues was 1.11.4 > The current version of our nifi instance is 1.15.3 > > > One other disadvantage of having Process Groups with Stopped components is > that when a dataflow manager or a developer inadvertently right click and > select “start” on the root canvas page, that will start all the components > in the child process groups. > > Thanks > > Deepak > > *[image: image005]* > > *Deepak Reddy* | Data Engineer > IT Centers of Excellence > 13736 Riverport Dr., Maryland Heights, MO 63043 > > > > *From:* Kevin Doran > *Sent:* Tuesday, December 13, 2022 10:26 AM > *To:* users@nifi.apache.org > *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry > > > > *CAUTION:* The e-mail below is from an external source. Please exercise > caution before opening attachments, clicking links, or following guidance. > > Thanks for sharing Deepak. That's interesting. I think we may look into a > way of disabling components that doesn't impact version control, or else > fix performance issues such that disabling is not necessary. > > > > Just out of curiosity, what version of NiFi are you on? The reason I ask > is that I know there have been some UI performance improvements made in > recent versions of NiFi, so I'm wondering if some of this might already > have been addressed... > > > > Thanks, > > Kevin > > > > On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" < > c-deepakreddy.chirth...@charter.com> wrote: > > Hi Kevin, > > Thank you for your quick response. The only reason why I prefer processors > to be disabled is to improve the NiFi UI performance by simply disabling > stopped processors which I am not using. > > > https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.cloudera.com%2Ft5%2FSupport-Questions%2FNifi-UI-Working-very-Slow-How-to-increase-performance-of-a%2Ftd-p%2F214550&data=05%7C01%7Cstephen.hindmarch%40bt.com%7C86e782f0c8b747d242e208dadebddd53%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C638067204299548928%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZVoWjub6gDAju1xyliaJXCZ%2BVcc5LF5NZn6%2FidKKe7s%3D&reserved=0> > > I had issues with Nifi UI performance in the past where the refresh wheel > keeps on spinning like for 10 seconds and disabling stopped processors >
Re: [EXTERNAL] Re: Disabling flows - nifi registry
Yeah I agree that disabling/enabling a processor should count as a change for the purpose of version control / nifi registry for the reasons Bryan and Stephen mentioned. I could see us introducing something like "pause tracking changes" which would change the icon for disabled PGs to something that just indicates it is disconnected from registry. And it would have to be "unpaused" prior to committing a new version. Would that be useful in this case or in general? Personally I don't mind having "local/dirty" state for a PG in a dev environment as I can always check local changes to see what is included, and IIRC, enabling a PG will remove the local changes icon. On Dec 15, 2022 at 12:58:19, Bryan Bende wrote: > Yea, the other side of this is when someone has disabled a portion of > their flow in a lower environment and committed to registry, and then > imports to their higher environment and expects that part to remain > disabled. Otherwise that change has to be redone in higher environment. > > On Thu, Dec 15, 2022 at 12:55 PM stephen.hindmarch.bt.com via users < > users@nifi.apache.org> wrote: > >> Personally I am against it. I feel that it is more important to protect >> production than development, and the operators running production will have >> less time and possibly less skill to look in depth at what should and >> should not be running, than the developers who are running development. >> >> >> >> My use case is that I want my operators to be able to press the run >> button without worrying about which processors or ports might need to be >> stopped. They should be able to observe the top level groups and see there >> are no stopped processors. >> >> >> >> As a developer, if I think a processor should not be run in production, >> then I will disable it before I commit it to the registry. These will >> include test injectors and fix/feedback loops that would be used often in >> development, but may also have a role in production during troubleshooting. >> The trigger to version control when the enable/disable state of a processor >> changes is an extra mechanism to warn if a processor has been used in >> production and has been accidentally left enabled. I prefer to review all >> changes in production before committing so that I am not accidentally >> committing untested “workarounds” that got me out of trouble this morning >> but would not like to persist in the code base. >> >> >> >> *Steve Hindmarch* >> >> >> >> *From:* Kevin Doran >> *Sent:* 15 December 2022 17:00 >> *To:* users@nifi.apache.org >> *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry >> >> >> >> Thanks, Deepak. I'll have to think about this one. I'm not sure if anyone >> else has any thoughts on this workflow / use case? >> >> >> >> On Dec 13, 2022 at 14:52:31, "Chirthani, Deepak Reddy" < >> c-deepakreddy.chirth...@charter.com> wrote: >> >> The version of Nifi where I had UI performance issues was 1.11.4 >> The current version of our nifi instance is 1.15.3 >> >> >> One other disadvantage of having Process Groups with Stopped components >> is that when a dataflow manager or a developer inadvertently right click >> and select “start” on the root canvas page, that will start all the >> components in the child process groups. >> >> Thanks >> >> Deepak >> >> *[image: image005]* >> >> *Deepak Reddy* | Data Engineer >> IT Centers of Excellence >> 13736 Riverport Dr., Maryland Heights, MO 63043 >> >> >> >> *From:* Kevin Doran >> *Sent:* Tuesday, December 13, 2022 10:26 AM >> *To:* users@nifi.apache.org >> *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry >> >> >> >> *CAUTION:* The e-mail below is from an external source. Please exercise >> caution before opening attachments, clicking links, or following guidance. >> >> Thanks for sharing Deepak. That's interesting. I think we may look into a >> way of disabling components that doesn't impact version control, or else >> fix performance issues such that disabling is not necessary. >> >> >> >> Just out of curiosity, what version of NiFi are you on? The reason I ask >> is that I know there have been some UI performance improvements made in >> recent versions of NiFi, so I'm wondering if some of this might already >> have been addressed... >> >> >> >> Thanks, >> >> Kevin >> >> >> >> O
RE: [EXTERNAL] Re: Disabling flows - nifi registry
Kevin, Out of Curiosity, has this been taken care in Nifi 1.20.0 which was released today? Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Chirthani, Deepak Reddy Sent: Tuesday, December 13, 2022 1:53 PM To: users@nifi.apache.org Subject: RE: [EXTERNAL] Re: Disabling flows - nifi registry The version of Nifi where I had UI performance issues was 1.11.4 The current version of our nifi instance is 1.15.3 One other disadvantage of having Process Groups with Stopped components is that when a dataflow manager or a developer inadvertently right click and select “start” on the root canvas page, that will start all the components in the child process groups. Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Kevin Doran mailto:kdo...@apache.org>> Sent: Tuesday, December 13, 2022 10:26 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: RE: [EXTERNAL] Re: Disabling flows - nifi registry CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Thanks for sharing Deepak. That's interesting. I think we may look into a way of disabling components that doesn't impact version control, or else fix performance issues such that disabling is not necessary. Just out of curiosity, what version of NiFi are you on? The reason I ask is that I know there have been some UI performance improvements made in recent versions of NiFi, so I'm wondering if some of this might already have been addressed... Thanks, Kevin On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" mailto:c-deepakreddy.chirth...@charter.com>> wrote: Hi Kevin, Thank you for your quick response. The only reason why I prefer processors to be disabled is to improve the NiFi UI performance by simply disabling stopped processors which I am not using. https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550 I had issues with Nifi UI performance in the past where the refresh wheel keeps on spinning like for 10 seconds and disabling stopped processors really helped me. Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Kevin Doran mailto:kdo...@apache.org>> Sent: Friday, December 9, 2022 9:33 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: [EXTERNAL] Re: Disabling flows - nifi registry CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi Deepak, So far, we have been honoring the following policy for what constitutes a change in version control: 1. stopped/started does not count as a "local change" 2. enabled/disabled does count as a change, and that state is captured in the flow snapshot json version saved to registry. One reason for this is that some users want to setup CI/CD to deploy from Registry and automatically start a flow in the target NiFi. If there are components they don't want to start, the disabled state gives them a way to capture that configuration in the flow. Is there a reason you prefer disable over stop in your lower environment? I leave flows stopped in dev environments all the time and have never run into an issue, but of course, everyone's workflow and use case is slightly different, so I'm interested in hearing your perspective on this to see if we need to consider something more flexible. Cheers, Kevin On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" mailto:c-deepakreddy.chirth...@charter.com>> wrote: Hey guys, So, once I fully develop and parameterize my nifi dataflow, let’s say in dev environment, I enable the version control, import the flow in higher environment and turn on the dataflow. In most of the cases both the flows in lower and higher environments will be running. Let says dev nifi connects to dev gcp pubsub and prod nifi connects to prod gcp pubsub. However, in some cases, we do want to stop and disable the flow in lower env. When I do that the registry is identifying that local changes are made to the flow which is nothing but all the components are disabled. I don’t want to keep the processors in stopped state on the canvas(registry do not identify for stopping) but want to disable them. Any workaround for registry not to identify local changes when flow is disabled? Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally
RE: [EXTERNAL] Re: Disabling flows - nifi registry
No, in 1.20.0, component enabled/disabled state is still treated as a change that affects versioning. Do let us know if you continue to see performance issues for your flow on recent versions of NiFi with all processors enabled. That is something that should be addressed. Regards, Kevin On Feb 9, 2023 at 21:38:46, "Chirthani, Deepak Reddy" < c-deepakreddy.chirth...@charter.com> wrote: > Kevin, > > > > Out of Curiosity, has this been taken care in Nifi 1.20.0 which was > released today? > > > > Thanks > > Deepak > > > > *[image: image005]* > > *Deepak Reddy* | Data Engineer > IT Centers of Excellence > 13736 Riverport Dr., Maryland Heights, MO 63043 > > > > *From:* Chirthani, Deepak Reddy > *Sent:* Tuesday, December 13, 2022 1:53 PM > *To:* users@nifi.apache.org > *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry > > > > The version of Nifi where I had UI performance issues was 1.11.4 > The current version of our nifi instance is 1.15.3 > > > One other disadvantage of having Process Groups with Stopped components is > that when a dataflow manager or a developer inadvertently right click and > select “start” on the root canvas page, that will start all the components > in the child process groups. > > Thanks > > Deepak > > *[image: image005]* > > *Deepak Reddy* | Data Engineer > IT Centers of Excellence > 13736 Riverport Dr., Maryland Heights, MO 63043 > > > > *From:* Kevin Doran > *Sent:* Tuesday, December 13, 2022 10:26 AM > *To:* users@nifi.apache.org > *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry > > > > *CAUTION:* The e-mail below is from an external source. Please exercise > caution before opening attachments, clicking links, or following guidance. > > Thanks for sharing Deepak. That's interesting. I think we may look into a > way of disabling components that doesn't impact version control, or else > fix performance issues such that disabling is not necessary. > > > > Just out of curiosity, what version of NiFi are you on? The reason I ask > is that I know there have been some UI performance improvements made in > recent versions of NiFi, so I'm wondering if some of this might already > have been addressed... > > > > Thanks, > > Kevin > > > > On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" < > c-deepakreddy.chirth...@charter.com> wrote: > > Hi Kevin, > > Thank you for your quick response. The only reason why I prefer processors > to be disabled is to improve the NiFi UI performance by simply disabling > stopped processors which I am not using. > > > https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550 > > I had issues with Nifi UI performance in the past where the refresh wheel > keeps on spinning like for 10 seconds and disabling stopped processors > really helped me. > > Thanks > Deepak > > > > > > *[image: image005]* > > *Deepak Reddy* | Data Engineer > IT Centers of Excellence > 13736 Riverport Dr., Maryland Heights, MO 63043 > > > > *From:* Kevin Doran > *Sent:* Friday, December 9, 2022 9:33 AM > *To:* users@nifi.apache.org > *Subject:* [EXTERNAL] Re: Disabling flows - nifi registry > > > > *CAUTION:* The e-mail below is from an external source. Please exercise > caution before opening attachments, clicking links, or following guidance. > > Hi Deepak, > > > > So far, we have been honoring the following policy for what constitutes a > change in version control: > > > >1. stopped/started does not count as a "local change" >2. enabled/disabled does count as a change, and that state is captured >in the flow snapshot json version saved to registry. > > > > One reason for this is that some users want to setup CI/CD to deploy from > Registry and automatically start a flow in the target NiFi. If there are > components they don't want to start, the disabled state gives them a way to > capture that configuration in the flow. > > > > Is there a reason you prefer disable over stop in your lower environment? > I leave flows stopped in dev environments all the time and have never run > into an issue, but of course, everyone's workflow and use case is slightly > different, so I'm interested in hearing your perspective on this to see if > we need to consider something more flexible. > > > > Cheers, > > Kevin > > > > On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" < > c-deepakreddy.chirth...@charter.com> wrote: > > Hey
RE: [EXTERNAL] Re: Disabling flows - nifi registry
Sure, will let you know if we face any UI performance issues. But for now, I will keep the dataflows/Process Groups in stopped state if we don’t need to turn them on in any environments unaffecting the version. Let me know if my thinking is incorrect. [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Kevin Doran Sent: Friday, February 10, 2023 7:24 AM To: users@nifi.apache.org Subject: RE: [EXTERNAL] Re: Disabling flows - nifi registry CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. No, in 1.20.0, component enabled/disabled state is still treated as a change that affects versioning. Do let us know if you continue to see performance issues for your flow on recent versions of NiFi with all processors enabled. That is something that should be addressed. Regards, Kevin On Feb 9, 2023 at 21:38:46, "Chirthani, Deepak Reddy" mailto:c-deepakreddy.chirth...@charter.com>> wrote: Kevin, Out of Curiosity, has this been taken care in Nifi 1.20.0 which was released today? Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Chirthani, Deepak Reddy Sent: Tuesday, December 13, 2022 1:53 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: RE: [EXTERNAL] Re: Disabling flows - nifi registry The version of Nifi where I had UI performance issues was 1.11.4 The current version of our nifi instance is 1.15.3 One other disadvantage of having Process Groups with Stopped components is that when a dataflow manager or a developer inadvertently right click and select “start” on the root canvas page, that will start all the components in the child process groups. Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Kevin Doran mailto:kdo...@apache.org>> Sent: Tuesday, December 13, 2022 10:26 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: RE: [EXTERNAL] Re: Disabling flows - nifi registry CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Thanks for sharing Deepak. That's interesting. I think we may look into a way of disabling components that doesn't impact version control, or else fix performance issues such that disabling is not necessary. Just out of curiosity, what version of NiFi are you on? The reason I ask is that I know there have been some UI performance improvements made in recent versions of NiFi, so I'm wondering if some of this might already have been addressed... Thanks, Kevin On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" mailto:c-deepakreddy.chirth...@charter.com>> wrote: Hi Kevin, Thank you for your quick response. The only reason why I prefer processors to be disabled is to improve the NiFi UI performance by simply disabling stopped processors which I am not using. https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550 I had issues with Nifi UI performance in the past where the refresh wheel keeps on spinning like for 10 seconds and disabling stopped processors really helped me. Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 From: Kevin Doran mailto:kdo...@apache.org>> Sent: Friday, December 9, 2022 9:33 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: [EXTERNAL] Re: Disabling flows - nifi registry CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi Deepak, So far, we have been honoring the following policy for what constitutes a change in version control: 1. stopped/started does not count as a "local change" 2. enabled/disabled does count as a change, and that state is captured in the flow snapshot json version saved to registry. One reason for this is that some users want to setup CI/CD to deploy from Registry and automatically start a flow in the target NiFi. If there are components they don't want to start, the disabled state gives them a way to capture that configuration in the flow. Is there a reason you prefer disable over stop in your lower environment? I leave flows stopped in dev environments all the time and have never run into an issue, but of course, everyone's workflow and use case is slightly different, so I'm interested in hearing your perspective on this to see if we need to consider something more flexible. Cheers, Kevin On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" mailto:c-deepakreddy.
