Do you have any controller services in process groups above the “controller container”?
I can’t remember if it is based on only the immediate parent, or the entire hierarchy. On Sat, May 23, 2020 at 8:00 PM Andrew Grande <apere...@gmail.com> wrote: > Maybe something is going on with specific types or hierarchies. I've > noticed DefaultSslContext didn't get assigned, even though it was the only > one available. Does autowiring logic apply to this one? > > Andrew > > On Sat, May 23, 2020, 3:54 PM Eric Secules <esecu...@gmail.com> wrote: > >> Hi Bryan, >> >> I have noticed this behaviour sometimes, but not all the time I am >> running the latest registry and NiFi versions. I haven't found a conclusive >> pattern but I have a hunch that it has to do with having versioned process >> groups within versioned process groups. My deployment strategy is this: >> >> - Have an outer process group which only contains controller >> services, called the "Controller Container" >> - For now I just have one controller service per type of >> controller service. >> - When deploying, download all production flows inside the >> Controller Container. >> - I noticed that some of the controller services find their match, >> but others don't leaving me with roughly 70 invalid processors out of 800. >> >> If you could point me in the right direction of the code which is >> supposed to do the matching I might be able to debug better. >> >> Thanks, >> Eric >> >> On Sat, May 23, 2020 at 3:27 PM Bryan Bende <bbe...@gmail.com> wrote: >> >>> If you use registry >= 0.5.0 And nifi >= 1.10.0, then it will auto >>> select external controller services with the same name as long as there is >>> only one of the same type with same name (name is not unique). >>> >>> On Sat, May 23, 2020 at 3:34 PM Andy LoPresto <alopre...@apache.org> >>> wrote: >>> >>>> My position is that we don’t claim completely automated deployment as a >>>> feature, so manually setting the controller service IDs is not exposed. >>>> Technically, they are defined in the flow.xml.gz and could be modified by >>>> an administrator to be static after generation. This would require frequent >>>> manual manipulation of the flow.xml.gz in various environments and frequent >>>> restarts of the NiFi service. I do not recommend this. >>>> >>>> >>>> Andy LoPresto >>>> alopre...@apache.org >>>> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* >>>> He/Him >>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>> >>>> On May 23, 2020, at 11:05 AM, Andrew Grande <apere...@gmail.com> wrote: >>>> >>>> Aren't those IDs generated? How can one enforce it? >>>> >>>> Andrew >>>> >>>> On Sat, May 23, 2020, 10:53 AM Andy LoPresto <alopre...@apache.org> >>>> wrote: >>>> >>>>> If you want the process to be completely automated, you would have to >>>>> enforce the controller service IDs to be identical across environments. >>>>> Otherwise deployment would need a manual intervention to reference the >>>>> specific controller service in the proper component. >>>>> >>>>> Andy LoPresto >>>>> alopre...@apache.org >>>>> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* >>>>> He/Him >>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>>> >>>>> On May 22, 2020, at 3:57 PM, Eric Secules <esecu...@gmail.com> wrote: >>>>> >>>>> Hi Andy, >>>>> >>>>> Given that you have a flow which operates on two different S3 accounts >>>>> for example, how would you do deployment automation? Do you mandate that >>>>> the controller service with the same ID must exist in both a development >>>>> and production environment rather than try to connect a processor to a >>>>> matching controller service? >>>>> >>>>> -Eric >>>>> >>>>> On Fri, May 22, 2020 at 3:44 PM Andy LoPresto <alopre...@apache.org> >>>>> wrote: >>>>> >>>>>> Eric, >>>>>> >>>>>> I can’t answer all these questions but I would definitely have >>>>>> hesitations around building an expectation that there is only one >>>>>> instance >>>>>> of any given controller service type in an entire canvas. I can think of >>>>>> numerous flows (this may not affect your particular flows, but the >>>>>> concepts >>>>>> still apply) which require multiple instances of the same controller >>>>>> service type to be available: >>>>>> >>>>>> * A flow which invokes a mutually-authenticated TLS HTTP API, >>>>>> consumes data, transforms it, and posts it to another mTLS API >>>>>> * A flow which retrieves objects from one S3 bucket and puts them >>>>>> into an S3 bucket in a different AWS account >>>>>> * A flow which connects to one database and retrieves data, >>>>>> transforms it, and persists it to another database >>>>>> >>>>>> If there is only _one_ StandardSSLContextService, >>>>>> AWSCredentialsProviderControllerService, or DBCPConnectionPool available >>>>>> in >>>>>> the entire controller, these flows cannot exist. >>>>>> >>>>>> I am not saying the retrieval of new flow versions and the matching >>>>>> of referenced controller services cannot be improved, but I would >>>>>> definitely advise caution before going too far down this path without >>>>>> considering all possible side effects and potential constraints on future >>>>>> flow development. >>>>>> >>>>>> >>>>>> Andy LoPresto >>>>>> alopre...@apache.org >>>>>> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* >>>>>> He/Him >>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>>>> >>>>>> On May 22, 2020, at 3:01 PM, Eric Secules <esecu...@gmail.com> wrote: >>>>>> >>>>>> Hello everyone, >>>>>> >>>>>> I am running into an issue with automated deployment using nipyapi >>>>>> <https://nipyapi.readthedocs.io/en/latest/>. We would like to be >>>>>> able to pull down flows from a registry and have them ready to go once >>>>>> all >>>>>> their controller services have been turned on. But there are a few >>>>>> issues. >>>>>> Sometimes the flows that we download from the registry reference >>>>>> controller >>>>>> service IDs that don't exist on this machine because the flow was >>>>>> developed >>>>>> in a different environment. That's easy enough to fix if there is just >>>>>> one >>>>>> applicable controller service, but not when there are two or more. >>>>>> >>>>>> We have taken the decision to put all our controller services at the >>>>>> top level and have one of each kind we need, rather than have multiple of >>>>>> the same controller service attached to individual process groups. >>>>>> >>>>>> We are running into a problem where some processors can either >>>>>> connect to a JSONTreeReader or a CSVReader and there's no indication in >>>>>> the >>>>>> ProcessorDTO object which type it was originally connected to, just a >>>>>> GUID >>>>>> of a controller service that doesn't exist in this deployment. >>>>>> >>>>>> Would it be possible to include the type or name of the controller >>>>>> service in the component.config.descriptors section? Are we going about >>>>>> it >>>>>> the wrong way trying to simplify down to the least number of controller >>>>>> services? >>>>>> >>>>>> Thanks, >>>>>> Eric >>>>>> >>>>>> >>>>>> >>>>> >>>> -- >>> Sent from Gmail Mobile >>> >> -- Sent from Gmail Mobile
