The short answer is - please revisit all of this after the 1.10.0 release because all of this will be solved with the new parameters feature :)
Longer answer... The CLI has two commands: nifi pg-get-vars nifi pg-set-var These can be used to get and set process group level variables, which can be referenced in any property that supports EL with at least VARIABLE_REGISTRY scope. Sensitive property values are never sent to registry and thus have to be set one time after first importing the flow from registry. This is what the discussion of using the REST API was for, since the CLI does not have a 'set-property' command to set a value of a single property on a component. On Tue, Sep 3, 2019 at 4:13 PM Evan Reynolds <e...@usermind.com.invalid> wrote: > > There is a CLI command to change variables ... ? I've moved most of my > processor variables to extra-args.property, so it's easy to change them. But > there are a few I can't change - for example, > DistributedMapCacheClientService controller service has a "Server Hostname" > property that doesn't support the expression language, so we have to > essentially search and replace the string in the flow.xml file before > deploying it. Similarly, the DBCPConnectionPool has a password that is a > sensitive values - so even if we set it to an argument, that argument gets > wiped when we go through the registry to deploy, so we are currently > deploying and then going and manually filling in that password. > > I just looked again at the CLI but I didn't see a command that would set > them, but I may have very easily missed it! Is there? > > --------------------------------------------------------------- > > Evan Reynolds > e...@usermind.com > > > On 9/2/19, 7:59 PM, "Andrew Grande" <apere...@gmail.com> wrote: > > I think there is confusion here. Changing *properties* is considered a > tracked change, the behavior is per design. > > There are *variables* on the PG level which are considered non-trackable > environment settings and won't trigger a dirty flag. > > There is a dedicated command in the cli to list and set these *variables*. > > Hope it helps, > Andrew > > On Mon, Sep 2, 2019, 4:07 PM sivasankar Reddy <mail2ms...@gmail.com> > wrote: > > > Hi Bryan, > > > > Thanks for the options. After pg-import or pg-change-version, if we use > > nifi rest api and update properties, looks like nifi instance detects > > changes(compares with bucket flow in registry) and creates * mark. > > > > We can change pg-change-version however just for properties setting , > may > > be its not advisable to change version in nifi instance. > > > > Any ideas around this will be helpful. > > > > Thanks > > Siva > > > > On 2019/08/23 16:23:38, Bryan Bende <bbe...@gmail.com> wrote: > > > Well there are two different encryption parts here - > > > encryption-at-rest and encryption-in-transit... > > > > > > Encryption-at-rest would have to be part of your script that is > > > calling the REST API. Somehow you create a config file with encrypted > > > values, and your script needs to read those in an decrypt them in > > > memory and then put them into the content that will be sent in the > > > HTTP PUT or POST request. > > > > > > Encryption-in-transit would be handled by the fact that you would be > > > making an HTTPS request to NiFi, so the contents of the POST or PUT > > > request are encrypted in transit. This is the same thing that is > > > happening if you were in the NiFi UI and typed in a password into the > > > sensitive property field. > > > > > > > > > On Fri, Aug 23, 2019 at 11:35 AM sivasankar Reddy > <mail2ms...@gmail.com> > > wrote: > > > > > > > > Hi Bryan, > > > > > > > > Thanks for the reply. > > > > Able to get the json object and set the property for Password. > However > > value of password can be either referred from config file (its plain > string > > visible in config). Is there any way that password is encrypted and this > > set property can take decrypted value. > > > > > > > > this way we ensure that password is sensitive. Any other ideas to > > achieve this > > > > > > > > > > > > Regards, > > > > Siva > > > > > > > > > > > > > > > > On 2019/08/22 17:38:37, Bryan Bende <bbe...@gmail.com> wrote: > > > > > The parameters work is on-going and will be in the next release > which > > > > > is 1.10.0. Releases don't really have specific timelines, but > > > > > generally they happen every few months. Most likely 1.10.0 is a > > couple > > > > > of weeks away, but depends on when active work is completed and > when > > > > > someone volunteers to make a release. > > > > > > > > > > There is no timeline for the "set-property" command since it was > just > > > > > suggested as a new feature in this email thread :) It requires > > someone > > > > > creating a JIRA and deciding to work on it. > > > > > > > > > > All of the functionality in the CLI and NiPyAPI, and even NiFi's > own > > > > > UI, is based on the REST API. So you can still perform a "set > > > > > property" by using the REST API to modify the configuration of a > > > > > processor, the CLI would just make it easier so that you wouldn't > > have > > > > > to understand the lower level API details. The best way to > understand > > > > > the API calls is to us the UI while you have Chrome Dev Tools > open, > > > > > and then perform the action you are interested in and look at the > > > > > requests made on the Network tab. You'll be able to see what URLs > are > > > > > called and what the request and response bodies look like. > > > > > > > > > > On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy < > > mail2ms...@gmail.com> wrote: > > > > > > > > > > > > Hi Bryan, > > > > > > > > > > > > Thanks for the reply. looks like even set-property will be a new > > feature, as current CLI doesn't have that. > > > > > > > > > > > > Could you please share timelines of these features if its in > > roadmap. > > > > > > 1. "set-property" > > > > > > 2. “parameters” > > > > > > > > > > > > The only option currently is set to set sensitive parameters > > manually? or any other option through CLI > > > > > > > > > > > > Regards, > > > > > > Siva > > > > > > > > > > > > On 2019/08/22 04:10:46, Bryan Bende <bbe...@gmail.com> wrote: > > > > > > > Currently there isn’t a good way with the CLI, we would need > to > > add a > > > > > > > command like set-property that took the id of a component, and > > the name and > > > > > > > value for a property. > > > > > > > > > > > > > > The next release will have a new feature called “parameters” > > which will > > > > > > > solve this problem. You’ll be able to use a parameter in a > > sensitive > > > > > > > property and then use CLI to set parameter values. > > > > > > > > > > > > > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy < > > mail2ms...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Bryan, > > > > > > > > > > > > > > > > Thanks for the update. Could you please mention about > > > > > > > > sensitive parameters aspect from deployment point of view > > using CLI. > > > > > > > > > > > > > > > > How to set the sensitive parameters values after pg-import. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Siva > > > > > > > > > > > > > > > > On 2019/08/21 11:47:49, Bryan Bende <bbe...@gmail.com> > wrote: > > > > > > > > > It does not, I was saying we could easily improve the > > command to add > > > > > > > > those > > > > > > > > > parameters. > > > > > > > > > > > > > > > > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy < > > mail2ms...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Bryan, > > > > > > > > > > > > > > > > > > > > Thanks for the update. Does the existing pg-import takes > > optional > > > > > > > > params > > > > > > > > > > for x/y coordinates. If yes can you share the sample > > command as > > > > > > > > pg-import > > > > > > > > > > --help doesn't has this info. > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > siva > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2019/08/20 22:19:23, Bryan Bende <bbe...@gmail.com> > > wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > 1) pg-import will keeping finding a location that does > > not overlap > > > > > > > > with > > > > > > > > > > any > > > > > > > > > > > existing process groups, so it is more of auto layout. > > We could > > > > > > > > easily > > > > > > > > > > add > > > > > > > > > > > optional params for x/y coordinates. > > > > > > > > > > > > > > > > > > > > > > 2) Both are fine choices.. CLI is more geared towards > > simple > > > > > > > > commands, > > > > > > > > > > > where as with NiPyApi you could more easily write > > complex logic in > > > > > > > > > > Python. > > > > > > > > > > > With CLI you would write a wrapper shell script that > > made individual > > > > > > > > > > calls > > > > > > > > > > > to the CLI. I think it is really just a matter of > > preference. > > > > > > > > > > > > > > > > > > > > > > -Bryan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy < > > > > > > > > mail2ms...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Bryan, > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your input. I would like to know more > > > > > > > > > > > > > > > > > > > > > > > > 1. how does CLI pg-import works if already canvas > has > > 100 to 150 > > > > > > > > > > flows. > > > > > > > > > > > > Will CLI pg-import always place the flow in canvas > > which can be > > > > > > > > easily > > > > > > > > > > > > located. As nipyapi has location(x,y) during deploy > > flow. > > > > > > > > > > > > > > > > > > > > > > > > 2. From enterprise usage, do you recommend toolkit > CLI > > vs nipyapi, > > > > > > > > it > > > > > > > > > > > > seems toolkit CLI is part of official Nifi. > > > > > > > > > > > > > > > > > > > > > > > > Any other points regarding this will be helpful. > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > Siva > > > > > > > > > > > > > > > > > > > > > > > > On 2019/08/08 22:41:43, Bryan Bende > <bbe...@gmail.com> > > wrote: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > A lot depends on whether you are doing the > promotion > > with the UI > > > > > > > > or > > > > > > > > > > > > > with scripts using something like NiFi CLI or > > NiPyApi. > > > > > > > > > > > > > > > > > > > > > > > > > > 1) Currently references to controller services > > outside the > > > > > > > > versioned > > > > > > > > > > > > > process group need to be set after initial import. > > In the next > > > > > > > > > > release > > > > > > > > > > > > > it will auto-select by name as long as there is > only > > one service > > > > > > > > with > > > > > > > > > > > > > the given name and type in a parent group (name is > > not unique). > > > > > > > > > > > > > > > > > > > > > > > > > > 2) The snapshot of the variables are part of the > > versioned flow, > > > > > > > > so > > > > > > > > > > > > > after importing you would have to set the new > values > > in the > > > > > > > > target > > > > > > > > > > > > > environment. This can be done in the UI or with > > scripts. > > > > > > > > > > > > > > > > > > > > > > > > > > 3) The REST API allows for positioning of any > > component that is > > > > > > > > > > > > > created. i can't speak for how to do it with > > NiPyApi, but I know > > > > > > > > the > > > > > > > > > > > > > CLI pg-import command attempts to automatically > > organize the > > > > > > > > imported > > > > > > > > > > > > > process groups on the root canvas so that they > don't > > overlap. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > Bryan > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy < > > > > > > > > > > mail2ms...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > We are looking for nifi deploy automation from > > regsitry -> > > > > > > > > higher > > > > > > > > > > nifi > > > > > > > > > > > > instance. Quick observations during intial > evaluation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. After process group import, controller > services > > are not set > > > > > > > > in > > > > > > > > > > nifi > > > > > > > > > > > > instance > > > > > > > > > > > > > > 2. How to set variables from source process > group > > to > > > > > > > > destination > > > > > > > > > > > > process group. > > > > > > > > > > > > > > 3. Is there any possibility of definining the > > positioning of > > > > > > > > > > process > > > > > > > > > > > > group to be imported in Nifi Canvas. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any other best practices regarding NiFi > > deployment automation > > > > > > > > > > will be > > > > > > > > > > > > helpful. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in Advance! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Sent from Gmail Mobile > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Sent from Gmail Mobile > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Sent from Gmail Mobile > > > > > > > > > > > > > > > > > >
