Re: How to integrate Secured Ragistry with Secured Nifi
Hi Mike, Kevin Thank you for your answers, I appreciate it. @Mike, why are you setting WEB_HTTP_HOST and WEB_HTTP_PORT when you are using secured nifi? Shouldnt that be empty and only HTTPS host and port used? BR, Tom On Mon, 18 Feb 2019, 23:56 Mike Thomsen Tom, > > > Note: both Registry and Nifi are in Docker containers on the same node. > I tried with IP address, but nothing. > > Each docker container has its own IP address. You need to link the two > containers. I always use Docker Compose, so I can't help you on how to set > it up manually. That said, I did a sample last year connecting a few > different NiFi nodes and a registry w/ SSL here: > > > https://github.com/MikeThomsen/nifi-docker-compose/blob/master/docker-compose-registry.yml > > I can't remember if I kept the LDAP docker container referenced in it, but > you should be able to look at it and figure out how to link everything up > from that with Docker Compose. > > Mike > > On Mon, Feb 18, 2019 at 12:00 PM Kevin Doran wrote: > >> >> Hi Tom, >> >> Given that you are getting a Connection refused exception and not an HTTP >> 401 or 403, I suspect that the problem is networking related and not >> authentication/authorization. >> >> Are the two docker containers on the same docker network? Can you >> resolve/ping the Registry container from the NiFi container, and when you >> create the Registry client in NiFi, are you using the hostname that the >> NiFi server/container would use to address Registry (ie, not the host a >> REgistry UI use might use if you are using port mapping to the docker >> container with the host). >> >> Here is an example repo in which I have an example of connecting NiFi and >> Registry and docker conatiners using docker-compose: >> >> https://github.com/kevdoran/flowops >> >> Hope this helps, >> Kevin >> >> >> On February 18, 2019 at 10:08:54, Tomislav Novosel (to.novo...@gmail.com) >> wrote: >> > Hi all, >> > >> > I generated standalone certificate with nifi-toolkit for my two Nifi >> > instances and for Nifi registry instance. All are on the same domain so >> I >> > used one certificate and its credentials for properties file (trustore >> > path, keystore path, keystore passw, trustore passw). >> > >> > Auth is configured through domain LDAP server and everything works. >> > >> > On both Nifi node and Registry node I configured authorizers.xml file on >> > property "Node Identity 1" with value from keystore.jks on "Owner" >> > attribute. >> > >> > Owner: <> >> > In Nifi registry I added that as user and gave rights to read and modify >> > buckets. >> > >> > When I add Registry Client on Nifi node and Hit Start version control on >> > process group I got error: >> > >> > Error >> > >> > Unable to obtain listing of buckets: java.net.ConnectException: >> Connection >> > refused (Connection refused) >> > >> > >> > I missed something in configuration, please help me. >> > >> > Note: both Registry and Nifi are in Docker containers on the same node. >> I >> > tried with IP address, but nothing. >> > >> > >> > Thank you, >> > >> > Tom >> > >> >>
Using variables in SSLContextService
Hello!
Also asked the question on IRC, but figured the mailing list might be better
for this longer question.
For an InvokeHTTP processor I defined a SSL Context Service. In the SSL Context
Service, in Keystore Filename property, I'd like to use a variable which I
defined in a nifi.registry file. So in my nifi.properties I have:
nifi.variable.registry.properties=./conf/custom.properties
And in the conf/custom.properties I have:
nifi.system.properties.file=C:/Users/some.file
And in the field Keystore Filename in the SSL Context Service I input:
${nifi.system.properties.file}
But then saving the SSL Context Service doesn't work anymore, the validation
fails and says "${nifi.system.properties.file} does not exist". The actual file
does exist however, but I suspect that the variable doesn't get interpolated.
According to
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-ssl-context-service-nar/1.5.0/org.apache.nifi.ssl.StandardSSLContextService/index.html
it seems like expression language is not supported for Keystore Filename
property. Does this also imply that variables won't work in that field?
Thanks for your help!
Max
===
Please access the attached hyperlink for an important electronic communications
disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===
Re: How to integrate Secured Ragistry with Secured Nifi
Tom, > Note: both Registry and Nifi are in Docker containers on the same node. I tried with IP address, but nothing. Each docker container has its own IP address. You need to link the two containers. I always use Docker Compose, so I can't help you on how to set it up manually. That said, I did a sample last year connecting a few different NiFi nodes and a registry w/ SSL here: https://github.com/MikeThomsen/nifi-docker-compose/blob/master/docker-compose-registry.yml I can't remember if I kept the LDAP docker container referenced in it, but you should be able to look at it and figure out how to link everything up from that with Docker Compose. Mike On Mon, Feb 18, 2019 at 12:00 PM Kevin Doran wrote: > > Hi Tom, > > Given that you are getting a Connection refused exception and not an HTTP > 401 or 403, I suspect that the problem is networking related and not > authentication/authorization. > > Are the two docker containers on the same docker network? Can you > resolve/ping the Registry container from the NiFi container, and when you > create the Registry client in NiFi, are you using the hostname that the > NiFi server/container would use to address Registry (ie, not the host a > REgistry UI use might use if you are using port mapping to the docker > container with the host). > > Here is an example repo in which I have an example of connecting NiFi and > Registry and docker conatiners using docker-compose: > > https://github.com/kevdoran/flowops > > Hope this helps, > Kevin > > > On February 18, 2019 at 10:08:54, Tomislav Novosel (to.novo...@gmail.com) > wrote: > > Hi all, > > > > I generated standalone certificate with nifi-toolkit for my two Nifi > > instances and for Nifi registry instance. All are on the same domain so I > > used one certificate and its credentials for properties file (trustore > > path, keystore path, keystore passw, trustore passw). > > > > Auth is configured through domain LDAP server and everything works. > > > > On both Nifi node and Registry node I configured authorizers.xml file on > > property "Node Identity 1" with value from keystore.jks on "Owner" > > attribute. > > > > Owner: <> > > In Nifi registry I added that as user and gave rights to read and modify > > buckets. > > > > When I add Registry Client on Nifi node and Hit Start version control on > > process group I got error: > > > > Error > > > > Unable to obtain listing of buckets: java.net.ConnectException: > Connection > > refused (Connection refused) > > > > > > I missed something in configuration, please help me. > > > > Note: both Registry and Nifi are in Docker containers on the same node. I > > tried with IP address, but nothing. > > > > > > Thank you, > > > > Tom > > > >
Re: How to integrate Secured Ragistry with Secured Nifi
Hi Tom, Given that you are getting a Connection refused exception and not an HTTP 401 or 403, I suspect that the problem is networking related and not authentication/authorization. Are the two docker containers on the same docker network? Can you resolve/ping the Registry container from the NiFi container, and when you create the Registry client in NiFi, are you using the hostname that the NiFi server/container would use to address Registry (ie, not the host a REgistry UI use might use if you are using port mapping to the docker container with the host). Here is an example repo in which I have an example of connecting NiFi and Registry and docker conatiners using docker-compose: https://github.com/kevdoran/flowops Hope this helps, Kevin On February 18, 2019 at 10:08:54, Tomislav Novosel (to.novo...@gmail.com) wrote: > Hi all, > > I generated standalone certificate with nifi-toolkit for my two Nifi > instances and for Nifi registry instance. All are on the same domain so I > used one certificate and its credentials for properties file (trustore > path, keystore path, keystore passw, trustore passw). > > Auth is configured through domain LDAP server and everything works. > > On both Nifi node and Registry node I configured authorizers.xml file on > property "Node Identity 1" with value from keystore.jks on "Owner" > attribute. > > Owner: <> > In Nifi registry I added that as user and gave rights to read and modify > buckets. > > When I add Registry Client on Nifi node and Hit Start version control on > process group I got error: > > Error > > Unable to obtain listing of buckets: java.net.ConnectException: Connection > refused (Connection refused) > > > I missed something in configuration, please help me. > > Note: both Registry and Nifi are in Docker containers on the same node. I > tried with IP address, but nothing. > > > Thank you, > > Tom >
How to integrate Secured Ragistry with Secured Nifi
Hi all, I generated standalone certificate with nifi-toolkit for my two Nifi instances and for Nifi registry instance. All are on the same domain so I used one certificate and its credentials for properties file (trustore path, keystore path, keystore passw, trustore passw). Auth is configured through domain LDAP server and everything works. On both Nifi node and Registry node I configured authorizers.xml file on property "Node Identity 1" with value from keystore.jks on "Owner" attribute. Owner: <
Re: Automate NiFi Ranger Policies
That's a good point, Chad. I don't know if there might be an easy way to get that info. The only thing I can think of is that when running with Apache Ranger, you have the option to enable Audit Log in which NiFi reports to Ranger any time an access policy is checked for authorization. I'm not sure if it would have the level of detail you need, but assuming it does, those audit logs should be accessible via the Ranger REST API, so when creating the policies for the Application PG, you could also check for a WRITE to the parent Ingest Channel PG, and try to correlate it to the creation of the Application PG based on a close timestamp? It seems a bit fiddly and hard to code up, but it could work... Others might chime in if they know of an easier way to get the creator/owner of a PG or component in general. We've talk a bit in the past about a new flow storage model that would store individual changes in sequence (to enable undo/redo on the canvas UI), and in that model we would have the author of each change. So this use case could be an additional motivation to work towards such a model. In the meantime, Ranger audit logs would be what I would look at. Regards, Kevin On February 15, 2019 at 14:35:01, Chad Woodhead (chadwoodh...@gmail.com) wrote: > Kevin, > > Thanks for the high level thought process. Seems like a feasible solution. > Do you know if I would be able to get the user who created the "Application > PG" to add them to the Ranger policy so they don't lose access to their own > application? Does NiFi keep that information? > > Thanks, > Chad > > On Fri, Feb 15, 2019 at 2:02 PM Kevin Doran wrote: > > > Hi Chad, > > > > I've never done this, but if I were to go about it I would create a > > script / cron job to poll the NiFi REST API [1] periodically, and upon > > detection of a new "Application PG", create the corresponding policies > > in Ranger via its REST API [2]. > > > > You'll have to create service accounts in both NiFI and Ranger for > > this script to run as and authenticate to both REST APIs, so you'll > > need a secure server to run it from that (1) has access to both NiFi > > and Ranger services and (2) has a way of restricting access to that > > management server, or at least the service account credentials that > > are stored on the server. And of course I would take care when > > automating access policy creation in any service! > > > > Tools like NiPyAPI [3][4] can help with scripting access to the NiFi > > REST API to poll for process groups. I'm not sure if a similar tool > > exists on the Ranger side, although they do have a published Swagger > > spec of their REST API [5][6], so generating something similar to > > NiPyApi using swagger-codegen [7] might be possible. Then again you > > only need to authenticate and access a few endpoints, so any scripting > > language with a decent HTTP client library should be sufficient for > > this type of thing. > > > > [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html > > [2] https://ranger.apache.org/apidocs/index.html > > [3] https://pypi.org/project/nipyapi/ > > [4] https://github.com/Chaffelson/nipyapi > > [5] https://ranger.apache.org/apidocs/ui/swagger.json > > [6] https://ranger.apache.org/apidocs/ui/index.html > > [7] https://swagger.io/tools/swagger-codegen/ > > > > Hope this helps, > > Kevin > > > > On February 15, 2019 at 13:11:27, Chad Woodhead (chadwoodh...@gmail.com) > > wrote: > > > We use Ranger with NiFi for security and we are looking to automate the > > > creation of our Ranger policies. > > > > > > The way we organize our flows is like this: > > > NiFi Root Canvas > Ingest Channel PG > Application PG > > > > > > We create 3 Ranger Policies per Application PG: > > > -/process-groups/ > > > -/data/process-groups/ > > > -/provenance-data/process-groups/ > > > > > > Admins create the Ingest Channel PGs and developers create the > > Application > > > PGs. We were thinking of automating any time a new Application PG is > > > created inside any of the Ingest Channel PGs, create the 3 corresponding > > > Ranger policies. Was curious to see if anyone else has implemented > > anything > > > like this and if so, any tips/suggestions of how to do it? > > > > > > Thanks, > > > Chad > > > > > >
Versioned flows not maintaining Load Balance Strategy
I am running NiFi 1.8.0 and NiFi Registry 0.3.0. I have noticed load balance strategies on queues aren't coming through versioned flows in NiFi Registry. Here are the steps I am performing: 1. Have existing flow running on latest flow version in Dev and Cert (flow has already been developed and in version control) 2. Add load balance strategy to queue in Dev flow 3. NiFi shows local changes and I commit the changes to NiFi Registry 4. Cert NIFi shows new flow version 5. Pull latest version down to Cert. The load balance strategy for the queue doesn't come with it. I then have to edit flow on Cert to add the load balance strategy for the queue which causes NiFi to see local changes which I then have to commit to Registry again but this time from Cert. I saw this JIRA https://issues.apache.org/jira/browse/NIFIREG-194 which made me think I shouldn't be experiencing the behavior I am seeing. Thanks, Chad
Re: Nifi provenance indexing throughput if it is being used as an event store
Sure. Thanks, Joe. On Sun, 17 Feb. 2019, 22:52 Joe Witt ali > > there are many variables here that are needed before anyone could know for > sure. > > but give it a try and measure amd forecast and youll know within a day or > two. > > thanks > > > On Sat, Feb 16, 2019, 11:37 PM Ali Nazemian >> Thanks, Joe. Given the fact that we would like to add a few attributes >> and set them to be indexed for the provenance, the mentioned rate should be >> alright? >> >> Cheers, >> Ali >> >> On Sat, Feb 16, 2019 at 2:56 PM Joe Witt wrote: >> >>> Ali >>> >>> You certainly can and at the rates you mention you should be able to >>> keep it for a good while. >>> >>> Just set the properties you need for your system and measure the rate at >>> which prov storage fills. >>> >>> Thanks >>> >>> On Fri, Feb 15, 2019 at 10:29 PM Ali Nazemian >>> wrote: >>> I didn't mean to use Nifi provenance search for an external provenance search. I meant to use it for internal search provenance but keep the provenance for a longer time than usual. It means instead of expecting it to keep provenance data for a few days, use it as an event store as it also provides the search capability. Regards, Ali On Sat, Feb 16, 2019 at 5:29 AM Andrew Grande wrote: > NiFi provenance searches are not a good integration pattern for > external systems. I.e. using it to periodicaly fetch history burdens the > cluster (those searches can be heavy) and disrupt normal processing SLAs. > > Pushing provenance events out to an external system (pitebtially even > filtered down to components of interest) is a much more predictable > pattern > and provides lots of flexibility on how to interpret the events. > > Andrew > > On Thu, Feb 14, 2019, 11:26 PM Ali Nazemian > wrote: > >> Can I expect the Nifi search provenance part do the job for me? >> >> On Fri, 15 Feb. 2019, 13:21 Mike Thomsen > wrote: >> >>> Ali, >>> >>> There is a site to site publishing task for provenance that you can >>> add as a root controller service that would be great here. It'll just >>> take >>> all of your provenance data periodically and ship it off to another NiFi >>> server or cluster that can process all of the provenance data as blocks >>> of >>> JSON data. A common pattern there is to filter down to the events you >>> want >>> and publish to ElasticSearch. >>> >>> On Thu, Feb 14, 2019 at 7:05 PM Ali Nazemian >>> wrote: >>> Hi All, I am investigating to see how Nifi provenance can be used as an event store for a long period of time. Our use case is very burst based and sometimes we may not receive any event for a period of time and sometimes we may get burst traffic. On average we can say maybe around 1000 eps is the expected throughput at this stage. Nifi has a powerful provenance that gives you an ability to also index based on some attributes. I am investigating how reliable is to use Nifi provenance store for a long period of time and enable index for a few extra attributes. Has anybody used Nifi provenance at this scale? Can lots of Lucene indices create other issues within Nifi as provenance uses Lucene for the indexing? P.S: Our use case is pretty light for Nifi as we are not going to have any ETL and Nifi is being used mostly as an Orchestrator of multiple Microservices. Regards, Ali >>> -- A.Nazemian >>> >> >> -- >> A.Nazemian >> >
