Re: nifi deploy automation - variables pass through (incl controller services)
Hi Bryan, Thanks for the options. After pg-import or pg-change-version, if we use nifi rest api and update properties, looks like nifi instance detects changes(compares with bucket flow in registry) and creates * mark. We can change pg-change-version however just for properties setting , may be its not advisable to change version in nifi instance. Any ideas around this will be helpful. Thanks Siva On 2019/08/23 16:23:38, Bryan Bende wrote: > Well there are two different encryption parts here - > encryption-at-rest and encryption-in-transit... > > Encryption-at-rest would have to be part of your script that is > calling the REST API. Somehow you create a config file with encrypted > values, and your script needs to read those in an decrypt them in > memory and then put them into the content that will be sent in the > HTTP PUT or POST request. > > Encryption-in-transit would be handled by the fact that you would be > making an HTTPS request to NiFi, so the contents of the POST or PUT > request are encrypted in transit. This is the same thing that is > happening if you were in the NiFi UI and typed in a password into the > sensitive property field. > > > On Fri, Aug 23, 2019 at 11:35 AM sivasankar Reddy > wrote: > > > > Hi Bryan, > > > > Thanks for the reply. > > Able to get the json object and set the property for Password. However > > value of password can be either referred from config file (its plain string > > visible in config). Is there any way that password is encrypted and this > > set property can take decrypted value. > > > > this way we ensure that password is sensitive. Any other ideas to achieve > > this > > > > > > Regards, > > Siva > > > > > > > > On 2019/08/22 17:38:37, Bryan Bende wrote: > > > The parameters work is on-going and will be in the next release which > > > is 1.10.0. Releases don't really have specific timelines, but > > > generally they happen every few months. Most likely 1.10.0 is a couple > > > of weeks away, but depends on when active work is completed and when > > > someone volunteers to make a release. > > > > > > There is no timeline for the "set-property" command since it was just > > > suggested as a new feature in this email thread :) It requires someone > > > creating a JIRA and deciding to work on it. > > > > > > All of the functionality in the CLI and NiPyAPI, and even NiFi's own > > > UI, is based on the REST API. So you can still perform a "set > > > property" by using the REST API to modify the configuration of a > > > processor, the CLI would just make it easier so that you wouldn't have > > > to understand the lower level API details. The best way to understand > > > the API calls is to us the UI while you have Chrome Dev Tools open, > > > and then perform the action you are interested in and look at the > > > requests made on the Network tab. You'll be able to see what URLs are > > > called and what the request and response bodies look like. > > > > > > On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy > > > wrote: > > > > > > > > Hi Bryan, > > > > > > > > Thanks for the reply. looks like even set-property will be a new > > > > feature, as current CLI doesn't have that. > > > > > > > > Could you please share timelines of these features if its in roadmap. > > > > 1. "set-property" > > > > 2. “parameters” > > > > > > > > The only option currently is set to set sensitive parameters manually? > > > > or any other option through CLI > > > > > > > > Regards, > > > > Siva > > > > > > > > On 2019/08/22 04:10:46, Bryan Bende wrote: > > > > > Currently there isn’t a good way with the CLI, we would need to add a > > > > > command like set-property that took the id of a component, and the > > > > > name and > > > > > value for a property. > > > > > > > > > > The next release will have a new feature called “parameters” which > > > > > will > > > > > solve this problem. You’ll be able to use a parameter in a sensitive > > > > > property and then use CLI to set parameter values. > > > > > > > > > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy > > > > > > > > > > wrote: > > > > > > > > > > > Hi Bry
Re: nifi deploy automation - variables pass through (incl controller services)
Hi Bryan, Thanks for the reply. Able to get the json object and set the property for Password. However value of password can be either referred from config file (its plain string visible in config). Is there any way that password is encrypted and this set property can take decrypted value. this way we ensure that password is sensitive. Any other ideas to achieve this Regards, Siva On 2019/08/22 17:38:37, Bryan Bende wrote: > The parameters work is on-going and will be in the next release which > is 1.10.0. Releases don't really have specific timelines, but > generally they happen every few months. Most likely 1.10.0 is a couple > of weeks away, but depends on when active work is completed and when > someone volunteers to make a release. > > There is no timeline for the "set-property" command since it was just > suggested as a new feature in this email thread :) It requires someone > creating a JIRA and deciding to work on it. > > All of the functionality in the CLI and NiPyAPI, and even NiFi's own > UI, is based on the REST API. So you can still perform a "set > property" by using the REST API to modify the configuration of a > processor, the CLI would just make it easier so that you wouldn't have > to understand the lower level API details. The best way to understand > the API calls is to us the UI while you have Chrome Dev Tools open, > and then perform the action you are interested in and look at the > requests made on the Network tab. You'll be able to see what URLs are > called and what the request and response bodies look like. > > On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy > wrote: > > > > Hi Bryan, > > > > Thanks for the reply. looks like even set-property will be a new feature, > > as current CLI doesn't have that. > > > > Could you please share timelines of these features if its in roadmap. > > 1. "set-property" > > 2. “parameters” > > > > The only option currently is set to set sensitive parameters manually? or > > any other option through CLI > > > > Regards, > > Siva > > > > On 2019/08/22 04:10:46, Bryan Bende wrote: > > > Currently there isn’t a good way with the CLI, we would need to add a > > > command like set-property that took the id of a component, and the name > > > and > > > value for a property. > > > > > > The next release will have a new feature called “parameters” which will > > > solve this problem. You’ll be able to use a parameter in a sensitive > > > property and then use CLI to set parameter values. > > > > > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy > > > wrote: > > > > > > > Hi Bryan, > > > > > > > > Thanks for the update. Could you please mention about > > > > sensitive parameters aspect from deployment point of view using CLI. > > > > > > > > How to set the sensitive parameters values after pg-import. > > > > > > > > Regards, > > > > Siva > > > > > > > > On 2019/08/21 11:47:49, Bryan Bende wrote: > > > > > It does not, I was saying we could easily improve the command to add > > > > those > > > > > parameters. > > > > > > > > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy > > > > > > > > > > wrote: > > > > > > > > > > > Hi Bryan, > > > > > > > > > > > > Thanks for the update. Does the existing pg-import takes optional > > > > params > > > > > > for x/y coordinates. If yes can you share the sample command as > > > > pg-import > > > > > > --help doesn't has this info. > > > > > > > > > > > > Regards, > > > > > > siva > > > > > > > > > > > > > > > > > > On 2019/08/20 22:19:23, Bryan Bende wrote: > > > > > > > Hello, > > > > > > > > > > > > > > 1) pg-import will keeping finding a location that does not overlap > > > > with > > > > > > any > > > > > > > existing process groups, so it is more of auto layout. We could > > > > easily > > > > > > add > > > > > > > optional params for x/y coordinates. > > > > > > > > > > > > > > 2) Both are fine choices.. CLI is more geared towards simple > > > > comman
Re: nifi deploy automation - variables pass through (incl controller services)
Hi Bryan, Thanks for the reply. looks like even set-property will be a new feature, as current CLI doesn't have that. Could you please share timelines of these features if its in roadmap. 1. "set-property" 2. “parameters” The only option currently is set to set sensitive parameters manually? or any other option through CLI Regards, Siva On 2019/08/22 04:10:46, Bryan Bende wrote: > Currently there isn’t a good way with the CLI, we would need to add a > command like set-property that took the id of a component, and the name and > value for a property. > > The next release will have a new feature called “parameters” which will > solve this problem. You’ll be able to use a parameter in a sensitive > property and then use CLI to set parameter values. > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy > wrote: > > > Hi Bryan, > > > > Thanks for the update. Could you please mention about > > sensitive parameters aspect from deployment point of view using CLI. > > > > How to set the sensitive parameters values after pg-import. > > > > Regards, > > Siva > > > > On 2019/08/21 11:47:49, Bryan Bende wrote: > > > It does not, I was saying we could easily improve the command to add > > those > > > parameters. > > > > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy > > > wrote: > > > > > > > Hi Bryan, > > > > > > > > Thanks for the update. Does the existing pg-import takes optional > > params > > > > for x/y coordinates. If yes can you share the sample command as > > pg-import > > > > --help doesn't has this info. > > > > > > > > Regards, > > > > siva > > > > > > > > > > > > On 2019/08/20 22:19:23, Bryan Bende wrote: > > > > > Hello, > > > > > > > > > > 1) pg-import will keeping finding a location that does not overlap > > with > > > > any > > > > > existing process groups, so it is more of auto layout. We could > > easily > > > > add > > > > > optional params for x/y coordinates. > > > > > > > > > > 2) Both are fine choices.. CLI is more geared towards simple > > commands, > > > > > where as with NiPyApi you could more easily write complex logic in > > > > Python. > > > > > With CLI you would write a wrapper shell script that made individual > > > > calls > > > > > to the CLI. I think it is really just a matter of preference. > > > > > > > > > > -Bryan > > > > > > > > > > > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy < > > mail2ms...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Bryan, > > > > > > > > > > > > Thanks for your input. I would like to know more > > > > > > > > > > > > 1. how does CLI pg-import works if already canvas has 100 to 150 > > > > flows. > > > > > > Will CLI pg-import always place the flow in canvas which can be > > easily > > > > > > located. As nipyapi has location(x,y) during deploy flow. > > > > > > > > > > > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi, > > it > > > > > > seems toolkit CLI is part of official Nifi. > > > > > > > > > > > > Any other points regarding this will be helpful. > > > > > > > > > > > > Regards, > > > > > > Siva > > > > > > > > > > > > On 2019/08/08 22:41:43, Bryan Bende wrote: > > > > > > > Hello, > > > > > > > > > > > > > > A lot depends on whether you are doing the promotion with the UI > > or > > > > > > > with scripts using something like NiFi CLI or NiPyApi. > > > > > > > > > > > > > > 1) Currently references to controller services outside the > > versioned > > > > > > > process group need to be set after initial import. In the next > > > > release > > > > > > > it will auto-select by name as long as there is only one service > > with > > > > > > > the given name and type in a parent group (name is not unique). > > > > > > > > > > > > > > 2) The snapshot of
Re: nifi deploy automation - variables pass through (incl controller services)
Hi Bryan, Thanks for the update. Could you please mention about sensitive parameters aspect from deployment point of view using CLI. How to set the sensitive parameters values after pg-import. Regards, Siva On 2019/08/21 11:47:49, Bryan Bende wrote: > It does not, I was saying we could easily improve the command to add those > parameters. > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy > wrote: > > > Hi Bryan, > > > > Thanks for the update. Does the existing pg-import takes optional params > > for x/y coordinates. If yes can you share the sample command as pg-import > > --help doesn't has this info. > > > > Regards, > > siva > > > > > > On 2019/08/20 22:19:23, Bryan Bende wrote: > > > Hello, > > > > > > 1) pg-import will keeping finding a location that does not overlap with > > any > > > existing process groups, so it is more of auto layout. We could easily > > add > > > optional params for x/y coordinates. > > > > > > 2) Both are fine choices.. CLI is more geared towards simple commands, > > > where as with NiPyApi you could more easily write complex logic in > > Python. > > > With CLI you would write a wrapper shell script that made individual > > calls > > > to the CLI. I think it is really just a matter of preference. > > > > > > -Bryan > > > > > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy > > > wrote: > > > > > > > Hi Bryan, > > > > > > > > Thanks for your input. I would like to know more > > > > > > > > 1. how does CLI pg-import works if already canvas has 100 to 150 > > flows. > > > > Will CLI pg-import always place the flow in canvas which can be easily > > > > located. As nipyapi has location(x,y) during deploy flow. > > > > > > > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi, it > > > > seems toolkit CLI is part of official Nifi. > > > > > > > > Any other points regarding this will be helpful. > > > > > > > > Regards, > > > > Siva > > > > > > > > On 2019/08/08 22:41:43, Bryan Bende wrote: > > > > > Hello, > > > > > > > > > > A lot depends on whether you are doing the promotion with the UI or > > > > > with scripts using something like NiFi CLI or NiPyApi. > > > > > > > > > > 1) Currently references to controller services outside the versioned > > > > > process group need to be set after initial import. In the next > > release > > > > > it will auto-select by name as long as there is only one service with > > > > > the given name and type in a parent group (name is not unique). > > > > > > > > > > 2) The snapshot of the variables are part of the versioned flow, so > > > > > after importing you would have to set the new values in the target > > > > > environment. This can be done in the UI or with scripts. > > > > > > > > > > 3) The REST API allows for positioning of any component that is > > > > > created. i can't speak for how to do it with NiPyApi, but I know the > > > > > CLI pg-import command attempts to automatically organize the imported > > > > > process groups on the root canvas so that they don't overlap. > > > > > > > > > > Thanks, > > > > > > > > > > Bryan > > > > > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy < > > mail2ms...@gmail.com> > > > > wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > We are looking for nifi deploy automation from regsitry -> higher > > nifi > > > > instance. Quick observations during intial evaluation. > > > > > > > > > > > > 1. After process group import, controller services are not set in > > nifi > > > > instance > > > > > > 2. How to set variables from source process group to destination > > > > process group. > > > > > > 3. Is there any possibility of definining the positioning of > > process > > > > group to be imported in Nifi Canvas. > > > > > > > > > > > > Any other best practices regarding NiFi deployment automation > > will be > > > > helpful. > > > > > > > > > > > > Thanks in Advance! > > > > > > > > > > > > > > > > > > -- > > > Sent from Gmail Mobile > > > > > > -- > Sent from Gmail Mobile >
Re: nifi deploy automation - variables pass through (incl controller services)
Hi Bryan, Thanks for the update. Does the existing pg-import takes optional params for x/y coordinates. If yes can you share the sample command as pg-import --help doesn't has this info. Regards, siva On 2019/08/20 22:19:23, Bryan Bende wrote: > Hello, > > 1) pg-import will keeping finding a location that does not overlap with any > existing process groups, so it is more of auto layout. We could easily add > optional params for x/y coordinates. > > 2) Both are fine choices.. CLI is more geared towards simple commands, > where as with NiPyApi you could more easily write complex logic in Python. > With CLI you would write a wrapper shell script that made individual calls > to the CLI. I think it is really just a matter of preference. > > -Bryan > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy > wrote: > > > Hi Bryan, > > > > Thanks for your input. I would like to know more > > > > 1. how does CLI pg-import works if already canvas has 100 to 150 flows. > > Will CLI pg-import always place the flow in canvas which can be easily > > located. As nipyapi has location(x,y) during deploy flow. > > > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi, it > > seems toolkit CLI is part of official Nifi. > > > > Any other points regarding this will be helpful. > > > > Regards, > > Siva > > > > On 2019/08/08 22:41:43, Bryan Bende wrote: > > > Hello, > > > > > > A lot depends on whether you are doing the promotion with the UI or > > > with scripts using something like NiFi CLI or NiPyApi. > > > > > > 1) Currently references to controller services outside the versioned > > > process group need to be set after initial import. In the next release > > > it will auto-select by name as long as there is only one service with > > > the given name and type in a parent group (name is not unique). > > > > > > 2) The snapshot of the variables are part of the versioned flow, so > > > after importing you would have to set the new values in the target > > > environment. This can be done in the UI or with scripts. > > > > > > 3) The REST API allows for positioning of any component that is > > > created. i can't speak for how to do it with NiPyApi, but I know the > > > CLI pg-import command attempts to automatically organize the imported > > > process groups on the root canvas so that they don't overlap. > > > > > > Thanks, > > > > > > Bryan > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy > > wrote: > > > > > > > > Hi, > > > > > > > > We are looking for nifi deploy automation from regsitry -> higher nifi > > instance. Quick observations during intial evaluation. > > > > > > > > 1. After process group import, controller services are not set in nifi > > instance > > > > 2. How to set variables from source process group to destination > > process group. > > > > 3. Is there any possibility of definining the positioning of process > > group to be imported in Nifi Canvas. > > > > > > > > Any other best practices regarding NiFi deployment automation will be > > helpful. > > > > > > > > Thanks in Advance! > > > > > > > > > > -- > Sent from Gmail Mobile >
Re: nifi deploy automation - variables pass through (incl controller services)
Hi Bryan, Thanks for your input. I would like to know more 1. how does CLI pg-import works if already canvas has 100 to 150 flows. Will CLI pg-import always place the flow in canvas which can be easily located. As nipyapi has location(x,y) during deploy flow. 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi, it seems toolkit CLI is part of official Nifi. Any other points regarding this will be helpful. Regards, Siva On 2019/08/08 22:41:43, Bryan Bende wrote: > Hello, > > A lot depends on whether you are doing the promotion with the UI or > with scripts using something like NiFi CLI or NiPyApi. > > 1) Currently references to controller services outside the versioned > process group need to be set after initial import. In the next release > it will auto-select by name as long as there is only one service with > the given name and type in a parent group (name is not unique). > > 2) The snapshot of the variables are part of the versioned flow, so > after importing you would have to set the new values in the target > environment. This can be done in the UI or with scripts. > > 3) The REST API allows for positioning of any component that is > created. i can't speak for how to do it with NiPyApi, but I know the > CLI pg-import command attempts to automatically organize the imported > process groups on the root canvas so that they don't overlap. > > Thanks, > > Bryan > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy wrote: > > > > Hi, > > > > We are looking for nifi deploy automation from regsitry -> higher nifi > > instance. Quick observations during intial evaluation. > > > > 1. After process group import, controller services are not set in nifi > > instance > > 2. How to set variables from source process group to destination process > > group. > > 3. Is there any possibility of definining the positioning of process group > > to be imported in Nifi Canvas. > > > > Any other best practices regarding NiFi deployment automation will be > > helpful. > > > > Thanks in Advance! > > >
nifi deploy automation - variables pass through (incl controller services)
Hi, We are looking for nifi deploy automation from regsitry -> higher nifi instance. Quick observations during intial evaluation. 1. After process group import, controller services are not set in nifi instance 2. How to set variables from source process group to destination process group. 3. Is there any possibility of definining the positioning of process group to be imported in Nifi Canvas. Any other best practices regarding NiFi deployment automation will be helpful. Thanks in Advance!
