Re: NiFi Repo's on Shared Storage
Rich I haven't experimented with NFS in a long time but early results with NFS were very uneven. Performance in general was fine but stability as it related to locking and other behaviors was less desirable. NiFi is no longer as aggressive with file locks as it used to be so it is possible things will go better nowadays. We dont typically recommend it to large scale users though. Network Attached Storage should be fine but NFS in particular is what i'm referring to. Thanks On Sat, Feb 9, 2019 at 9:02 PM Rich T wrote: > Has anyone successfully (or unsuccessfully) implemented a NiFi cluster > where they NFS mounted the Flow/Prov/Content repo's from an NFS share to > the NiFi nodes? > > I am investigating installing a NIFI Cluster on a group of servers with > minimal local storage but high-performance CPUs. I also have access to a > High-Performance Flash NAS with high-bandwidth network interconnect, which > I can use for the Flow/Provenance/Content repos. My use case is a fairly > high dataflow rate. On paper, the NAS appears to be able to support the > IO/latency necessary to support processing the flows. > I would prefer to not be the first to do this, so if anyone has done this, > I would greatly appreciate hearing of successes or issues. >
NiFi Repo's on Shared Storage
Has anyone successfully (or unsuccessfully) implemented a NiFi cluster where they NFS mounted the Flow/Prov/Content repo's from an NFS share to the NiFi nodes? I am investigating installing a NIFI Cluster on a group of servers with minimal local storage but high-performance CPUs. I also have access to a High-Performance Flash NAS with high-bandwidth network interconnect, which I can use for the Flow/Provenance/Content repos. My use case is a fairly high dataflow rate. On paper, the NAS appears to be able to support the IO/latency necessary to support processing the flows. I would prefer to not be the first to do this, so if anyone has done this, I would greatly appreciate hearing of successes or issues.
Re: Record-oriented DetectDuplicate?
PR if anyone is interested: https://github.com/apache/nifi/pull/3298 On Fri, Feb 8, 2019 at 5:34 PM Mike Thomsen wrote: > With Redis and HBase you can set a TTL on the data itself in the lookup > table. Were you thinking something more than that? > > On Fri, Feb 8, 2019 at 4:42 PM Andrew Grande wrote: > >> Can I suggest a time-based option for specifying the window? I think we >> only mentioned the number of records. >> >> Andrew >> >> On Fri, Feb 8, 2019, 8:22 AM Mike Thomsen wrote: >> >>> Thanks. That answers it succinctly for me. I'll build out a >>> DetectDuplicateRecord processor to handle this. >>> >>> On Fri, Feb 8, 2019 at 11:17 AM Mark Payne wrote: >>> Matt, That would work if you want to select distinct records in a given FlowFIle but not across FlowFiles. PartitionRecord -> UpdateAttribute (optionally to combine multiple attributes into one) -> DetectDuplicate would work, but given that you expect the records to be unique generally, this would have the effect of splitting each FlowFile into Record-per-FlowFile, which is certainly not ideal. Thanks -Mark > On Feb 8, 2019, at 11:14 AM, Matt Burgess wrote: > > Mike, > > I don't think so, but you could try a SELECT DISTINCT in QueryRecord, > might be a bit of a pain if you want to select all columns and there > are lots of them. > > Alternatively you could try PartitionRecord -> QueryRecord (select * > limit 1). Neither PartitionRecord nor QueryRecord keeps state so you'd > likely need to use distributed cache or UpdateAttribute. > > Regards, > Matt > > On Fri, Feb 8, 2019 at 11:08 AM Mike Thomsen wrote: >> >> Do we have anything like DetectDuplicate for the Record API already? Didn't see anything, but wanted to ask before reinventing the wheel. >> >> Thanks, >> >> Mike
Re: Nifi registry Kerberos Auth with Docker
Yes, this log INFO I see also on my Nifi registry without docker.
I configured logback to DEBUG inside my container and I found in app.log
this exception:
un.security.krb5.KrbException: Cannot locate default realm
I started container with --add-host command to add in hosts file all my
Domain Kerberos FQDN's
for KDC server. I tried also to ping it from container and it is alive.
FYI. I crated volume for container to have access to krb5.conf file.
krb5.conf file is the same as for
my Nifi registry as a service where everything works. File permissions are
added also.
Don't know what else it could be.
Any suggestions?
Thank you.
BR,
Tom
On Fri, 8 Feb 2019 at 16:03, Bryan Bende wrote:
> Thinking about it more, I guess if you are not trying to do spnego
> then that message from the logs is not really an error. The registry
> UI always tries the spnego end-point first and if it returns the
> conflict response (as the log says) then you get sent to the login
> page.
>
> Maybe try turning on debug logging by editing logback.xml name="org.apache.nifi.registry" level="INFO"/> and changing to DEBUG.
>
> On Fri, Feb 8, 2019 at 9:51 AM Tomislav Novosel
> wrote:
> >
> > Hi Bryan,
> >
> > I don't have this properties populated in Nifi registry instance
> > outside Docker (as a service on linux server), and everything works.
> >
> > What are this properties up to?
> >
> > Regards,
> > Tom
> >
> >
> >
> > On Fri, 8 Feb 2019 at 15:25, Bryan Bende wrote:
> >>
> >> The message about "Kerberos service ticket login not supported by this
> >> NiFi Registry" means that one of the following properties is not
> >> populated:
> >>
> >> nifi.registry.kerberos.spnego.principal=
> >> nifi.registry.kerberos.spnego.keytab.location=
> >>
> >> On Fri, Feb 8, 2019 at 8:20 AM Tomislav Novosel
> wrote:
> >> >
> >> > Hi Daniel,
> >> >
> >> > Ok, I see. Thanks for the answer.
> >> >
> >> > I switched to official Nifi registry image. I succeeded to spin up
> registry in docker container and to
> >> > setup Kerberos provider in identity-providers.xml. Also I configured
> authorizers.xml as per afficial Nifi documentation.
> >> >
> >> > I already have the same setup with Kerberos, but not in Docker
> container. And everything works like a charm.
> >> >
> >> > When I enter credentials, login does not pass. This is app log:
> >> >
> >> > 2019-02-08 12:52:30,568 INFO [NiFi Registry Web Server-14]
> o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Kerberos service ticket login not supported by this NiFi Registry.
> Returning Conflict response.
> >> > 2019-02-08 12:52:30,644 INFO [NiFi Registry Web Server-13]
> o.a.n.r.w.s.NiFiRegistrySecurityConfig Client could not be authenticated
> due to:
> org.springframework.security.authentication.AuthenticationCredentialsNotFoundException:
> An Authentication object was not found in the SecurityContext Returning 401
> response.
> >> > 2019-02-08 12:52:50,557 INFO [NiFi Registry Web Server-14]
> o.a.n.r.w.m.UnauthorizedExceptionMapper
> org.apache.nifi.registry.web.exception.UnauthorizedException: The supplied
> client credentials are not valid.. Returning Unauthorized response.
> >> >
> >> > Not sure what is going on here.
> >> >
> >> > Regards,
> >> > Tom
> >> >
> >> >
> >> > On Fri, 8 Feb 2019 at 11:36, Daniel Chaffelson
> wrote:
> >> >>
> >> >> Hi Tomislav,
> >> >> I created that build a long time ago before the official apache one
> was up, and it is out of date sorry.
> >> >> Can I suggest you switch to the official apache image that Kevin
> mentioned and try again? It is an up to date version and recommended by the
> community.
> >> >>
> >> >> On Thu, Feb 7, 2019 at 5:54 PM Tomislav Novosel <
> to.novo...@gmail.com> wrote:
> >> >>>
> >> >>> Hi Kevin,
> >> >>>
> >> >>> I'm using image from Docker hub on this link:
> >> >>> https://hub.docker.com/r/chaffelson/nifi-registry
> >> >>>
> >> >>> I think I know where is the problem. The problem is in config file
> where
> >> >>> http host and http port property remains even if I manually set
> https host and htpps port.
> >> >>> I deleted http host and http port to be empty, but when I started
> container again, those values are again there.
> >> >>>
> >> >>> I don't know what the author of image wanted to say with this:
> >> >>>
> >> >>> The Docker image can be built using the following command:
> >> >>>
> >> >>> .
> ~/Projects/nifi-dev/nifi-registry/nifi-registry-docker/dockerhub/DockerBuild.sh
> >> >>>
> >> >>> What does this commend mean?
> >> >>>
> >> >>> And this:
> >> >>>
> >> >>> Note: The default version of NiFi-Registry specified by the
> Dockerfile is typically that of one that is unreleased if working from
> source. To build an image for a prior released version, one can override
> the NIFI_REGISTRY_VERSIONbuild-arg with the following command:
> >> >>>
> >> >>> docker build --build-arg=NIFI_REGISRTY_VERSION={Desired
> NiFi-Registry Version} -t apache/nifi-registry:latest .
> >> >>>
> >> >>> For this command
Re: Nifi registry Kerberos Auth with Docker
Yes, this log INFO I see also on my Nifi registry without docker.
I configured logback to DEBUG inside my container and I found in app.log
this exeption:
un.security.krb5.KrbException: Cannot locate default realm
I started container with --ad-host command to add in hosts file all my
Domain Kerberos FQDN's
for KDC server. I triad also to ping it from container and it is alive.
Don't know what else it could be.
Any suggestions?
Thank you.
BR,
Tom
On Fri, 8 Feb 2019 at 16:03, Bryan Bende wrote:
> Thinking about it more, I guess if you are not trying to do spnego
> then that message from the logs is not really an error. The registry
> UI always tries the spnego end-point first and if it returns the
> conflict response (as the log says) then you get sent to the login
> page.
>
> Maybe try turning on debug logging by editing logback.xml name="org.apache.nifi.registry" level="INFO"/> and changing to DEBUG.
>
> On Fri, Feb 8, 2019 at 9:51 AM Tomislav Novosel
> wrote:
> >
> > Hi Bryan,
> >
> > I don't have this properties populated in Nifi registry instance
> > outside Docker (as a service on linux server), and everything works.
> >
> > What are this properties up to?
> >
> > Regards,
> > Tom
> >
> >
> >
> > On Fri, 8 Feb 2019 at 15:25, Bryan Bende wrote:
> >>
> >> The message about "Kerberos service ticket login not supported by this
> >> NiFi Registry" means that one of the following properties is not
> >> populated:
> >>
> >> nifi.registry.kerberos.spnego.principal=
> >> nifi.registry.kerberos.spnego.keytab.location=
> >>
> >> On Fri, Feb 8, 2019 at 8:20 AM Tomislav Novosel
> wrote:
> >> >
> >> > Hi Daniel,
> >> >
> >> > Ok, I see. Thanks for the answer.
> >> >
> >> > I switched to official Nifi registry image. I succeeded to spin up
> registry in docker container and to
> >> > setup Kerberos provider in identity-providers.xml. Also I configured
> authorizers.xml as per afficial Nifi documentation.
> >> >
> >> > I already have the same setup with Kerberos, but not in Docker
> container. And everything works like a charm.
> >> >
> >> > When I enter credentials, login does not pass. This is app log:
> >> >
> >> > 2019-02-08 12:52:30,568 INFO [NiFi Registry Web Server-14]
> o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Kerberos service ticket login not supported by this NiFi Registry.
> Returning Conflict response.
> >> > 2019-02-08 12:52:30,644 INFO [NiFi Registry Web Server-13]
> o.a.n.r.w.s.NiFiRegistrySecurityConfig Client could not be authenticated
> due to:
> org.springframework.security.authentication.AuthenticationCredentialsNotFoundException:
> An Authentication object was not found in the SecurityContext Returning 401
> response.
> >> > 2019-02-08 12:52:50,557 INFO [NiFi Registry Web Server-14]
> o.a.n.r.w.m.UnauthorizedExceptionMapper
> org.apache.nifi.registry.web.exception.UnauthorizedException: The supplied
> client credentials are not valid.. Returning Unauthorized response.
> >> >
> >> > Not sure what is going on here.
> >> >
> >> > Regards,
> >> > Tom
> >> >
> >> >
> >> > On Fri, 8 Feb 2019 at 11:36, Daniel Chaffelson
> wrote:
> >> >>
> >> >> Hi Tomislav,
> >> >> I created that build a long time ago before the official apache one
> was up, and it is out of date sorry.
> >> >> Can I suggest you switch to the official apache image that Kevin
> mentioned and try again? It is an up to date version and recommended by the
> community.
> >> >>
> >> >> On Thu, Feb 7, 2019 at 5:54 PM Tomislav Novosel <
> to.novo...@gmail.com> wrote:
> >> >>>
> >> >>> Hi Kevin,
> >> >>>
> >> >>> I'm using image from Docker hub on this link:
> >> >>> https://hub.docker.com/r/chaffelson/nifi-registry
> >> >>>
> >> >>> I think I know where is the problem. The problem is in config file
> where
> >> >>> http host and http port property remains even if I manually set
> https host and htpps port.
> >> >>> I deleted http host and http port to be empty, but when I started
> container again, those values are again there.
> >> >>>
> >> >>> I don't know what the author of image wanted to say with this:
> >> >>>
> >> >>> The Docker image can be built using the following command:
> >> >>>
> >> >>> .
> ~/Projects/nifi-dev/nifi-registry/nifi-registry-docker/dockerhub/DockerBuild.sh
> >> >>>
> >> >>> What does this commend mean?
> >> >>>
> >> >>> And this:
> >> >>>
> >> >>> Note: The default version of NiFi-Registry specified by the
> Dockerfile is typically that of one that is unreleased if working from
> source. To build an image for a prior released version, one can override
> the NIFI_REGISTRY_VERSIONbuild-arg with the following command:
> >> >>>
> >> >>> docker build --build-arg=NIFI_REGISRTY_VERSION={Desired
> NiFi-Registry Version} -t apache/nifi-registry:latest .
> >> >>>
> >> >>> For this command above you need to have Dockerfile. I tried with
> Dockerfile from docker hub, but there are errors in execution on this line:
> >> >>>
> >> >>> ADD sh/ ${NIFI_REGISTRY_BASE_DIR}/scripts/
> >>
