Re: Migrating NiFi templates/canvas
Hello Matt, NIFI-3173 is exactly what I am referring to. We are currently using 1.0.0 but will upgrade to 1.1.1 to get the fix [😊] Many thanks, Panos From: Matt Gilman Sent: Friday, January 13, 2017 8:16 PM To: users@nifi.apache.org Subject: Re: Migrating NiFi templates/canvas Panos, I'm not sure what version of NiFi you're using but in 1.1.1 another issue [1] was addressed that may be what you're seeing now. If you're still on 1.0.0 or 1.1.0 maybe try upgrading to 1.1.1. Matt [1] https://issues.apache.org/jira/browse/NIFI-3173 On Fri, Jan 13, 2017 at 1:51 PM, Panos Geo mailto:panospanos1...@outlook.com>> wrote: Hello Matt, Thanks for your reply. I fully get that, but my requirement is in addition to this. The (top) controller service is being referenced by multiple processors in my template, which however belong in different sub process groups. When I import this template in another NiFi instance this controller service is present, but it is being "split" to different process group controller services, which I cannot configure centrally (as I would have done if that was a top level controller service as it was in the original NiFi instance before exporting the template). Many thanks, Panos From: Matt Gilman mailto:matt.c.gil...@gmail.com>> Sent: Friday, January 13, 2017 6:27 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Migrating NiFi templates/canvas Panos, The current behavior is that Controller Services are only included if they are referenced by a component in the template. While this was initially by designed, we have received comments about it from folks looking to export their entire configuration. There is an outstanding JIRA [1] to update this behavior. Matt [1] https://issues.apache.org/jira/browse/NIFI-2895 On Fri, Jan 13, 2017 at 12:01 PM, Panos Geo mailto:panospanos1...@outlook.com>> wrote: Having just checked the idea in my PS, it doesn't seem to work as the transferred controller services are at a process group level and hence are "compartmentalised" after the migration... Any thoughts on this would be highly appreciated. Many thanks, Panos From: Panos Geo mailto:panospanos1...@outlook.com>> Sent: Friday, January 13, 2017 4:47 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Migrating NiFi templates/canvas Hello Oleg, Many thanks for your reply! I remembered having diff'ed NiFi templates in the past and that was a bit messy. However, this was certainly before NiFi 1.0, so from a quick look it appears that things are better now in that regard. Thanks for the hint! However, the situation with moving the controller services associated with a processor seems still problematic in my humble opinion. Let's say that in my source VM, I have created a Database and an HTTPContext map service at the top level of my canvas. These are being used inside many different processors that belong to different process groups. Now, if I export my canvas as a template and import it on another VM, I don't see these services at the top level of my canvas on my target VM. The services are being "compartmentalised" into the process groups and sub process groups of my template. Therefore, if I have to do a change (say the port of the database is different) I need to go through all of them manually as a change in a "compartmentalised" service is not reflected on another process group. An alternative, which I have been using, is to recreate the service at the top level of the canvas on the target VM, but this means that I need to go through all the processors of the template I moved and associate them with the new top level service of my target VM, which is very error-prone in an CI/CD pipeline. Isn't there an easier way to migrate templates with their associated services? Many thanks again, Panos Ps. writing all this, it came to mind that I could include all my canvas in one process group in my source VM, generate my service inside that and then move all of that in my target VM. Not ideal, but should work [😉] From: Oleg Zhurakousky mailto:ozhurakou...@hortonworks.com>> Sent: Friday, January 13, 2017 1:18 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Migrating NiFi templates/canvas Panos What version of NiFi are you using? The issue with the NiFi templates diffs has actually been addressed (see https://issues.apache.org/jira/browse/NIFI-826). So as of NiFi 1.0 you can export templates in a deterministic way and then execute diff on them to see only what was changed. It was addressed specifically for the SDLC issue you are describing. Also, when processor is associated with controller ser
Re: Migrating NiFi templates/canvas
Panos, I'm not sure what version of NiFi you're using but in 1.1.1 another issue [1] was addressed that may be what you're seeing now. If you're still on 1.0.0 or 1.1.0 maybe try upgrading to 1.1.1. Matt [1] https://issues.apache.org/jira/browse/NIFI-3173 On Fri, Jan 13, 2017 at 1:51 PM, Panos Geo wrote: > Hello Matt, > > > Thanks for your reply. > > > I fully get that, but my requirement is in addition to this. The > (top) controller service is being referenced by multiple processors in my > template, which however belong in different sub process groups. When I > import this template in another NiFi instance this controller service is > present, but it is being "split" to different process group controller > services, which I cannot configure centrally (as I would have done if that > was a top level controller service as it was in the original NiFi instance > before exporting the template). > > > Many thanks, > > Panos > > > -- > *From:* Matt Gilman > *Sent:* Friday, January 13, 2017 6:27 PM > > *To:* users@nifi.apache.org > *Subject:* Re: Migrating NiFi templates/canvas > > Panos, > > The current behavior is that Controller Services are only included if they > are referenced by a component in the template. While this was initially by > designed, we have received comments about it from folks looking to export > their entire configuration. There is an outstanding JIRA [1] to update this > behavior. > > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-2895 > > On Fri, Jan 13, 2017 at 12:01 PM, Panos Geo > wrote: > >> Having just checked the idea in my PS, it doesn't seem to work as the >> transferred controller services are at a process group level and hence are " >> compartmentalised" after the migration... >> >> >> Any thoughts on this would be highly appreciated. >> >> >> Many thanks, >> >> Panos >> >> >> -- >> *From:* Panos Geo >> *Sent:* Friday, January 13, 2017 4:47 PM >> >> *To:* users@nifi.apache.org >> *Subject:* Re: Migrating NiFi templates/canvas >> >> >> Hello Oleg, >> >> >> Many thanks for your reply! >> >> >> I remembered having diff'ed NiFi templates in the past and that was a >> bit messy. However, this was certainly before NiFi 1.0, so from a quick >> look it appears that things are better now in that regard. Thanks for >> the hint! >> >> >> However, the situation with moving the controller services associated >> with a processor seems still problematic in my humble opinion. Let's say >> that in my source VM, I have created a Database and an HTTPContext map >> service at the *top level* of my canvas. These are being used inside >> many different processors that belong to different process groups. Now, >> if I export my canvas as a template and import it on another VM, I don't >> see these services at the *top level* of my canvas on my target VM. The >> services are being "compartmentalised" into the process groups and sub >> process groups of my template. Therefore, if I have to do a change (say >> the port of the database is different) I need to go through all of them >> manually as a change in a "compartmentalised" service is not reflected >> on another process group. >> >> >> An alternative, which I have been using, is to recreate the service at >> the top level of the canvas on the target VM, but this means that I need to >> go through all the processors of the template I moved and associate them >> with the new top level service of my target VM, which is very error-prone >> in an CI/CD pipeline. >> >> >> Isn't there an easier way to migrate templates with their associated >> services? >> >> >> Many thanks again, >> >> Panos >> >> >> >> Ps. writing all this, it came to mind that I could include all my canvas >> in one process group in my source VM, generate my service inside that and >> then move all of that in my target VM. Not ideal, but should work [image: >> 😉] >> >> >> -- >> *From:* Oleg Zhurakousky >> *Sent:* Friday, January 13, 2017 1:18 PM >> *To:* users@nifi.apache.org >> *Subject:* Re: Migrating NiFi templates/canvas >> >> Panos >> >> What version of NiFi are you using? >> The issue with the NiFi templates diffs has actually been addressed (see >> https://issues.apache.org/jira/browse/NI
Re: Migrating NiFi templates/canvas
Hello Matt, Thanks for your reply. I fully get that, but my requirement is in addition to this. The (top) controller service is being referenced by multiple processors in my template, which however belong in different sub process groups. When I import this template in another NiFi instance this controller service is present, but it is being "split" to different process group controller services, which I cannot configure centrally (as I would have done if that was a top level controller service as it was in the original NiFi instance before exporting the template). Many thanks, Panos From: Matt Gilman Sent: Friday, January 13, 2017 6:27 PM To: users@nifi.apache.org Subject: Re: Migrating NiFi templates/canvas Panos, The current behavior is that Controller Services are only included if they are referenced by a component in the template. While this was initially by designed, we have received comments about it from folks looking to export their entire configuration. There is an outstanding JIRA [1] to update this behavior. Matt [1] https://issues.apache.org/jira/browse/NIFI-2895 On Fri, Jan 13, 2017 at 12:01 PM, Panos Geo mailto:panospanos1...@outlook.com>> wrote: Having just checked the idea in my PS, it doesn't seem to work as the transferred controller services are at a process group level and hence are "compartmentalised" after the migration... Any thoughts on this would be highly appreciated. Many thanks, Panos From: Panos Geo mailto:panospanos1...@outlook.com>> Sent: Friday, January 13, 2017 4:47 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Migrating NiFi templates/canvas Hello Oleg, Many thanks for your reply! I remembered having diff'ed NiFi templates in the past and that was a bit messy. However, this was certainly before NiFi 1.0, so from a quick look it appears that things are better now in that regard. Thanks for the hint! However, the situation with moving the controller services associated with a processor seems still problematic in my humble opinion. Let's say that in my source VM, I have created a Database and an HTTPContext map service at the top level of my canvas. These are being used inside many different processors that belong to different process groups. Now, if I export my canvas as a template and import it on another VM, I don't see these services at the top level of my canvas on my target VM. The services are being "compartmentalised" into the process groups and sub process groups of my template. Therefore, if I have to do a change (say the port of the database is different) I need to go through all of them manually as a change in a "compartmentalised" service is not reflected on another process group. An alternative, which I have been using, is to recreate the service at the top level of the canvas on the target VM, but this means that I need to go through all the processors of the template I moved and associate them with the new top level service of my target VM, which is very error-prone in an CI/CD pipeline. Isn't there an easier way to migrate templates with their associated services? Many thanks again, Panos Ps. writing all this, it came to mind that I could include all my canvas in one process group in my source VM, generate my service inside that and then move all of that in my target VM. Not ideal, but should work [😉] From: Oleg Zhurakousky mailto:ozhurakou...@hortonworks.com>> Sent: Friday, January 13, 2017 1:18 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Migrating NiFi templates/canvas Panos What version of NiFi are you using? The issue with the NiFi templates diffs has actually been addressed (see https://issues.apache.org/jira/browse/NIFI-826). So as of NiFi 1.0 you can export templates in a deterministic way and then execute diff on them to see only what was changed. It was addressed specifically for the SDLC issue you are describing. Also, when processor is associated with controller service and you created a template that includes such processor, its dependent controller service will end up on the template and back in your environment once such template is imported. [NIFI-826] Export templates in a deterministic way - ASF JIRA<https://issues.apache.org/jira/browse/NIFI-826> issues.apache.org<http://issues.apache.org> Templates should be exported in a deterministic way so that they can be compared or diff'ed with another. Items to consider... The ordering of components The id's ... Anyway, give it a shot and let us know. Cheers Oleg On Jan 13, 2017, at 4:49 AM, Panos Geo mailto:panospanos1...@outlook.com>> wrote: Hello all, We have a Continuous Integration/Continuous Development pipeline that involve
Re: Migrating NiFi templates/canvas
Panos, The current behavior is that Controller Services are only included if they are referenced by a component in the template. While this was initially by designed, we have received comments about it from folks looking to export their entire configuration. There is an outstanding JIRA [1] to update this behavior. Matt [1] https://issues.apache.org/jira/browse/NIFI-2895 On Fri, Jan 13, 2017 at 12:01 PM, Panos Geo wrote: > Having just checked the idea in my PS, it doesn't seem to work as the > transferred controller services are at a process group level and hence are " > compartmentalised" after the migration... > > > Any thoughts on this would be highly appreciated. > > > Many thanks, > > Panos > > > -- > *From:* Panos Geo > *Sent:* Friday, January 13, 2017 4:47 PM > > *To:* users@nifi.apache.org > *Subject:* Re: Migrating NiFi templates/canvas > > > Hello Oleg, > > > Many thanks for your reply! > > > I remembered having diff'ed NiFi templates in the past and that was a bit > messy. However, this was certainly before NiFi 1.0, so from a quick look > it appears that things are better now in that regard. Thanks for the hint! > > > However, the situation with moving the controller services associated with > a processor seems still problematic in my humble opinion. Let's say that > in my source VM, I have created a Database and an HTTPContext map service > at the *top level* of my canvas. These are being used inside many > different processors that belong to different process groups. Now, if I > export my canvas as a template and import it on another VM, I don't see > these services at the *top level* of my canvas on my target VM. The > services are being "compartmentalised" into the process groups and sub > process groups of my template. Therefore, if I have to do a change (say > the port of the database is different) I need to go through all of them > manually as a change in a "compartmentalised" service is not reflected on > another process group. > > > An alternative, which I have been using, is to recreate the service at > the top level of the canvas on the target VM, but this means that I need to > go through all the processors of the template I moved and associate them > with the new top level service of my target VM, which is very error-prone > in an CI/CD pipeline. > > > Isn't there an easier way to migrate templates with their associated > services? > > > Many thanks again, > > Panos > > > > Ps. writing all this, it came to mind that I could include all my canvas > in one process group in my source VM, generate my service inside that and > then move all of that in my target VM. Not ideal, but should work [image: > 😉] > > > -- > *From:* Oleg Zhurakousky > *Sent:* Friday, January 13, 2017 1:18 PM > *To:* users@nifi.apache.org > *Subject:* Re: Migrating NiFi templates/canvas > > Panos > > What version of NiFi are you using? > The issue with the NiFi templates diffs has actually been addressed (see > https://issues.apache.org/jira/browse/NIFI-826). So as of NiFi 1.0 you > can export templates in a deterministic way and then execute diff on them > to see only what was changed. It was addressed specifically for the SDLC > issue you are describing. Also, when processor is associated with > controller service and you created a template that includes such processor, > its dependent controller service will end up on the template and back in > your environment once such template is imported. > [NIFI-826] Export templates in a deterministic way - ASF JIRA > <https://issues.apache.org/jira/browse/NIFI-826> > issues.apache.org > Templates should be exported in a deterministic way so that they can be > compared or diff'ed with another. Items to consider... The ordering of > components The id's ... > > > Anyway, give it a shot and let us know. > Cheers > Oleg > > On Jan 13, 2017, at 4:49 AM, Panos Geo wrote: > > Hello all, > > We have a Continuous Integration/Continuous Development pipeline that > involves for each stage (development, testing, customer deployment) a > dedicated virtual machine running a NiFi instance. As such, for each stage > of the pipeline we have to migrate the changes of our NiFi canvas from one > virtual machine to another. > > For this we encounter two big problems : > >1. As far as I know, there is no easy way to diff NiFi canvases (or >templates) so that we can recognize the changes and don’t have to >export and import the full canvas each time (as there are configuration >changes in the target VM that we wouldn’t like to overwrite). >2. If we have to export the full canvas, is there a way to reassociate >processors with the required services on the target virtual machine (e.g. >DBCPConnectionPool or HTTPContextMap) without having to go through each >processor explicitly and do the reassignment? > > > Many thanks in advance, > Panos > > >
Re: Migrating NiFi templates/canvas
Having just checked the idea in my PS, it doesn't seem to work as the transferred controller services are at a process group level and hence are "compartmentalised" after the migration... Any thoughts on this would be highly appreciated. Many thanks, Panos From: Panos Geo Sent: Friday, January 13, 2017 4:47 PM To: users@nifi.apache.org Subject: Re: Migrating NiFi templates/canvas Hello Oleg, Many thanks for your reply! I remembered having diff'ed NiFi templates in the past and that was a bit messy. However, this was certainly before NiFi 1.0, so from a quick look it appears that things are better now in that regard. Thanks for the hint! However, the situation with moving the controller services associated with a processor seems still problematic in my humble opinion. Let's say that in my source VM, I have created a Database and an HTTPContext map service at the top level of my canvas. These are being used inside many different processors that belong to different process groups. Now, if I export my canvas as a template and import it on another VM, I don't see these services at the top level of my canvas on my target VM. The services are being "compartmentalised" into the process groups and sub process groups of my template. Therefore, if I have to do a change (say the port of the database is different) I need to go through all of them manually as a change in a "compartmentalised" service is not reflected on another process group. An alternative, which I have been using, is to recreate the service at the top level of the canvas on the target VM, but this means that I need to go through all the processors of the template I moved and associate them with the new top level service of my target VM, which is very error-prone in an CI/CD pipeline. Isn't there an easier way to migrate templates with their associated services? Many thanks again, Panos Ps. writing all this, it came to mind that I could include all my canvas in one process group in my source VM, generate my service inside that and then move all of that in my target VM. Not ideal, but should work [😉] From: Oleg Zhurakousky Sent: Friday, January 13, 2017 1:18 PM To: users@nifi.apache.org Subject: Re: Migrating NiFi templates/canvas Panos What version of NiFi are you using? The issue with the NiFi templates diffs has actually been addressed (see https://issues.apache.org/jira/browse/NIFI-826). So as of NiFi 1.0 you can export templates in a deterministic way and then execute diff on them to see only what was changed. It was addressed specifically for the SDLC issue you are describing. Also, when processor is associated with controller service and you created a template that includes such processor, its dependent controller service will end up on the template and back in your environment once such template is imported. [NIFI-826] Export templates in a deterministic way - ASF JIRA<https://issues.apache.org/jira/browse/NIFI-826> issues.apache.org Templates should be exported in a deterministic way so that they can be compared or diff'ed with another. Items to consider... The ordering of components The id's ... Anyway, give it a shot and let us know. Cheers Oleg On Jan 13, 2017, at 4:49 AM, Panos Geo mailto:panospanos1...@outlook.com>> wrote: Hello all, We have a Continuous Integration/Continuous Development pipeline that involves for each stage (development, testing, customer deployment) a dedicated virtual machine running a NiFi instance. As such, for each stage of the pipeline we have to migrate the changes of our NiFi canvas from one virtual machine to another. For this we encounter two big problems : 1. As far as I know, there is no easy way to diff NiFi canvases (or templates) so that we can recognize the changes and don’t have to export and import the full canvas each time (as there are configuration changes in the target VM that we wouldn’t like to overwrite). 2. If we have to export the full canvas, is there a way to reassociate processors with the required services on the target virtual machine (e.g. DBCPConnectionPool or HTTPContextMap) without having to go through each processor explicitly and do the reassignment? Many thanks in advance, Panos
Re: Migrating NiFi templates/canvas
Hello Oleg, Many thanks for your reply! I remembered having diff'ed NiFi templates in the past and that was a bit messy. However, this was certainly before NiFi 1.0, so from a quick look it appears that things are better now in that regard. Thanks for the hint! However, the situation with moving the controller services associated with a processor seems still problematic in my humble opinion. Let's say that in my source VM, I have created a Database and an HTTPContext map service at the top level of my canvas. These are being used inside many different processors that belong to different process groups. Now, if I export my canvas as a template and import it on another VM, I don't see these services at the top level of my canvas on my target VM. The services are being "compartmentalised" into the process groups and sub process groups of my template. Therefore, if I have to do a change (say the port of the database is different) I need to go through all of them manually as a change in a "compartmentalised" service is not reflected on another process group. An alternative, which I have been using, is to recreate the service at the top level of the canvas on the target VM, but this means that I need to go through all the processors of the template I moved and associate them with the new top level service of my target VM, which is very error-prone in an CI/CD pipeline. Isn't there an easier way to migrate templates with their associated services? Many thanks again, Panos Ps. writing all this, it came to mind that I could include all my canvas in one process group in my source VM, generate my service inside that and then move all of that in my target VM. Not ideal, but should work [😉] From: Oleg Zhurakousky Sent: Friday, January 13, 2017 1:18 PM To: users@nifi.apache.org Subject: Re: Migrating NiFi templates/canvas Panos What version of NiFi are you using? The issue with the NiFi templates diffs has actually been addressed (see https://issues.apache.org/jira/browse/NIFI-826). So as of NiFi 1.0 you can export templates in a deterministic way and then execute diff on them to see only what was changed. It was addressed specifically for the SDLC issue you are describing. Also, when processor is associated with controller service and you created a template that includes such processor, its dependent controller service will end up on the template and back in your environment once such template is imported. [NIFI-826] Export templates in a deterministic way - ASF JIRA<https://issues.apache.org/jira/browse/NIFI-826> issues.apache.org Templates should be exported in a deterministic way so that they can be compared or diff'ed with another. Items to consider... The ordering of components The id's ... Anyway, give it a shot and let us know. Cheers Oleg On Jan 13, 2017, at 4:49 AM, Panos Geo mailto:panospanos1...@outlook.com>> wrote: Hello all, We have a Continuous Integration/Continuous Development pipeline that involves for each stage (development, testing, customer deployment) a dedicated virtual machine running a NiFi instance. As such, for each stage of the pipeline we have to migrate the changes of our NiFi canvas from one virtual machine to another. For this we encounter two big problems : 1. As far as I know, there is no easy way to diff NiFi canvases (or templates) so that we can recognize the changes and don’t have to export and import the full canvas each time (as there are configuration changes in the target VM that we wouldn’t like to overwrite). 2. If we have to export the full canvas, is there a way to reassociate processors with the required services on the target virtual machine (e.g. DBCPConnectionPool or HTTPContextMap) without having to go through each processor explicitly and do the reassignment? Many thanks in advance, Panos
Re: Migrating NiFi templates/canvas
Panos What version of NiFi are you using? The issue with the NiFi templates diffs has actually been addressed (see https://issues.apache.org/jira/browse/NIFI-826). So as of NiFi 1.0 you can export templates in a deterministic way and then execute diff on them to see only what was changed. It was addressed specifically for the SDLC issue you are describing. Also, when processor is associated with controller service and you created a template that includes such processor, its dependent controller service will end up on the template and back in your environment once such template is imported. Anyway, give it a shot and let us know. Cheers Oleg On Jan 13, 2017, at 4:49 AM, Panos Geo mailto:panospanos1...@outlook.com>> wrote: Hello all, We have a Continuous Integration/Continuous Development pipeline that involves for each stage (development, testing, customer deployment) a dedicated virtual machine running a NiFi instance. As such, for each stage of the pipeline we have to migrate the changes of our NiFi canvas from one virtual machine to another. For this we encounter two big problems : 1. As far as I know, there is no easy way to diff NiFi canvases (or templates) so that we can recognize the changes and don’t have to export and import the full canvas each time (as there are configuration changes in the target VM that we wouldn’t like to overwrite). 2. If we have to export the full canvas, is there a way to reassociate processors with the required services on the target virtual machine (e.g. DBCPConnectionPool or HTTPContextMap) without having to go through each processor explicitly and do the reassignment? Many thanks in advance, Panos
Migrating NiFi templates/canvas
Hello all, We have a Continuous Integration/Continuous Development pipeline that involves for each stage (development, testing, customer deployment) a dedicated virtual machine running a NiFi instance. As such, for each stage of the pipeline we have to migrate the changes of our NiFi canvas from one virtual machine to another. For this we encounter two big problems : 1. As far as I know, there is no easy way to diff NiFi canvases (or templates) so that we can recognize the changes and don't have to export and import the full canvas each time (as there are configuration changes in the target VM that we wouldn't like to overwrite). 2. If we have to export the full canvas, is there a way to reassociate processors with the required services on the target virtual machine (e.g. DBCPConnectionPool or HTTPContextMap) without having to go through each processor explicitly and do the reassignment? Many thanks in advance, Panos
