Re: GRPC processors missing after upgrade to 1.11.3
Dirk Apologies on the grpc bundle being included. Just a mistake on my part during the RM process. Thanks Joe On Mon, Mar 9, 2020 at 9:18 AM Bryan Bende wrote: > Hello, > > We likely need Joe to confirm about grpc since he was the RM for the last > few releases, but I believe that it was probably just a mistake in the > build process that caused it to not be excluded. > > For the other ones you referenced, those were purposely excluded due to > space limitations with the binary artifacts that we publish. It was > mentioned in the migration guidance that is created for every release [1]. > > "We've removed the following nars from the default convenience binary. > These include kite-nar, kafka-0-8-nar, flume-nar, media-nar, > druid-controller-service-api-nar, druid-nar, other-graph-services-nar. You > can still get them from the various artifact repositories and use them in > your flows but we cannot bundle them due to space limitations by default." > > Thanks, > > Bryan > > [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance > > On Mon, Mar 9, 2020 at 8:48 AM Dirk Arends > wrote: > >> Hey, >> >> I recently upgraded from 1.11.1 to 1.11.3 and found that the GRPC >> processors (ListenGRPC and InvokeGRPC) had seemingly been removed from the >> packaged release. But it seems that the source code in the Nifi repo for >> the “nifi-grpc-bundle” is still there and has not been removed. >> >> I have a couple of different versions in use and have found the bundles >> are missing in some but not others: >> >> - 1.11.0: GRPC nar is missing >> - 1.11.1: GRPC nar is included >> - 1.11.3: GRPC nar is missing >> >> My suspicion is the package was accidentally not included in the packaged >> release, is that plausible? >> >> This reminded me of another processor (ConvertJSONToAvro) which during >> the 1.9.x to 1.10.0 upgrade was removed without warning. At the time I put >> it down to the Nifi community preferring the use of “ConvertRecord” >> instead. Again, I found that the source code remained in the repo and was >> referenced in the “nifi-nar-bundles/pom.xml”. >> >> These links add to the confusion. While nifi-kite-nar seems to have been >> removed after 1.9.x, nifi-grpc-nar reports to not be included/referenced >> since 1.4, even while I have been using the processors through to 1.11.1. >> >> https://mvnrepository.com/artifact/org.apache.nifi/nifi-kite-nar >> https://mvnrepository.com/artifact/org.apache.nifi/nifi-grpc-nar >> >> There have not been any commits in the last few months affecting these >> bundles aside from release preparation commits. >> >> I have been able to copy the 1.11.1 version and run that on 1.11.3 for >> now, but I am hoping it can be included again in the next release. >> >> Thanks in advance, >> >> -- >> Dirk Arends >> >
Re: NiFi Registry tarball modes
Hello, It looks like most of the file modes in NiFi Registry were set to 0660, where as NiFi sets them to 0664. I created this JIRA [1] and submitted a PR [2]. -Bryan [1] https://issues.apache.org/jira/browse/NIFIREG-369 [2] https://github.com/apache/nifi-registry/pull/265 On Mon, Mar 9, 2020 at 5:11 AM Johannes J. Meixner wrote: > Moin, > > The jars inside of nifi-registry-0.5.0-bin.tar.gz are owned by > bbende:staff with mode 0660. > > In a future version of the tarball, would you mind chmodding the files > in lib/ and subdirectories to 0644, so I don't have to? > > Thanks > Johannes > >
Re: GRPC processors missing after upgrade to 1.11.3
Hello, We likely need Joe to confirm about grpc since he was the RM for the last few releases, but I believe that it was probably just a mistake in the build process that caused it to not be excluded. For the other ones you referenced, those were purposely excluded due to space limitations with the binary artifacts that we publish. It was mentioned in the migration guidance that is created for every release [1]. "We've removed the following nars from the default convenience binary. These include kite-nar, kafka-0-8-nar, flume-nar, media-nar, druid-controller-service-api-nar, druid-nar, other-graph-services-nar. You can still get them from the various artifact repositories and use them in your flows but we cannot bundle them due to space limitations by default." Thanks, Bryan [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance On Mon, Mar 9, 2020 at 8:48 AM Dirk Arends wrote: > Hey, > > I recently upgraded from 1.11.1 to 1.11.3 and found that the GRPC > processors (ListenGRPC and InvokeGRPC) had seemingly been removed from the > packaged release. But it seems that the source code in the Nifi repo for > the “nifi-grpc-bundle” is still there and has not been removed. > > I have a couple of different versions in use and have found the bundles > are missing in some but not others: > > - 1.11.0: GRPC nar is missing > - 1.11.1: GRPC nar is included > - 1.11.3: GRPC nar is missing > > My suspicion is the package was accidentally not included in the packaged > release, is that plausible? > > This reminded me of another processor (ConvertJSONToAvro) which during the > 1.9.x to 1.10.0 upgrade was removed without warning. At the time I put it > down to the Nifi community preferring the use of “ConvertRecord” instead. > Again, I found that the source code remained in the repo and was referenced > in the “nifi-nar-bundles/pom.xml”. > > These links add to the confusion. While nifi-kite-nar seems to have been > removed after 1.9.x, nifi-grpc-nar reports to not be included/referenced > since 1.4, even while I have been using the processors through to 1.11.1. > > https://mvnrepository.com/artifact/org.apache.nifi/nifi-kite-nar > https://mvnrepository.com/artifact/org.apache.nifi/nifi-grpc-nar > > There have not been any commits in the last few months affecting these > bundles aside from release preparation commits. > > I have been able to copy the 1.11.1 version and run that on 1.11.3 for > now, but I am hoping it can be included again in the next release. > > Thanks in advance, > > -- > Dirk Arends >
Re: Exception is showing in nifi UI users page
There is no file that contains this mapping, the users and groups are periodically loaded from LDAP into memory on each nifi node. The id of each user and group is a UUID that is seeded from the corresponding name. UUID.nameUUIDFromBytes(seed.getBytes(StandardCharsets.UTF_8)).toString(); For a group, "seed" would be the group name. On Mon, Mar 9, 2020 at 5:05 AM sanjeet rath wrote: > Hi team, > > I have the AD group name from the authoriser.xml file i want to delete the > corresponding Unique group dientifier value from authorisation.xml file. > > Could you please help me to find out the file where we are keeping the > relation map between AD group name and corresponding Unique group idetifier > value. > > Thanks a lot, > Sanjeet > > On Fri, 6 Mar 2020, 8:07 pm sanjeet rath, wrote: > >> Yes Mat, >> >> But i cant upgrading nifi till next 2 months. >> As i am not able to add user so its a blocker. >> >> Thats why asking u how the AD group name from the authoriser file is >> mapped to group identifier of authorization.xml file. >> >> Any place where we are storing this maping? >> >> So that i can delete it from authorization.xml >> >> Any other workaround is there. >> >> Thanks, >> Sanjeet >> >> On Fri, 6 Mar 2020, 7:24 pm Matt Gilman, wrote: >> >>> I would recommend upgrading. If I recall correctly, there were actually >>> a couple of JIRAs addressing different but similar issues dealing with >>> users or groups being removed from a directory server. Even if you could >>> identify and remove the problematic entries, you could possibly hit this >>> again if another user or group were removed later. >>> >>> Thanks >>> >>> On Fri, Mar 6, 2020 at 6:35 AM sanjeet rath >>> wrote: >>> Hi , The AD group names are present in Authoriser.xml file, In authorisation.xml file the group identifier (unique id) is present inside policy identifier tag. So could you please help me to understand where AD group name in authoriser.xml file and group identifier in authorisation.xml map together. So that i can delete it in authoriser.xml file. Thanks, Sanjeet On Fri, Mar 6, 2020 at 12:04 AM sanjeet rath wrote: > Thanks Mat, > > For this quick response. > > Thanks a lot , > > Sanjeet > > On Thu, 5 Mar 2020, 11:43 pm Matt Gilman, > wrote: > >> I just responded to your StackOverflow post: >> >> >> https://stackoverflow.com/questions/60551242/nifi-user-addition-gives-u-null-pointer-exception/60551638#60551638 >> >> I believe you'll need to upgrade to a version that addresses the BUG. >> >> Thanks! >> >> On Thu, Mar 5, 2020 at 1:10 PM sanjeet rath >> wrote: >> >>> Hi Team, >>> >>> I am using nifi cluster from 1 month & i am able to add new user >>> policies everything.its a LDAP based user addition. >>> but suddenly from last 2 days , in nifi user addition page(after >>> clicking on users in nifi UI) i am getting Error message "An unexcepted >>> error has occure.please click logs for more details". >>> and in nifif-user.log i found the bellow log. >>> >>> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has >>> occurred: java.lang.NullPointerException. Returning Internal Server >>> Error >>> response. java.lang.NullPointerException: null at >>> org.apache.nifi.web.dao.impl.StandardPolicyBasedAuthorizerDAO.lambda$null$2(StandardPolicyBasedAuthorizerDAO.java:285) >>> at >>> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) >>> at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553) >>> at >>> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at >>> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) >>> at >>> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) >>> at >>> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at >>> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) >>> >>> I am not able to figuring out where should i start looking out. >>> Could please someone help me to look a starting point where can >>> check and what need to be checked. >>> >>> Thanks, >>> Sanjeet >>> >>> >>> -- >>> Sanjeet Kumar Rath, >>> mob- +91 8777577470 >>> >>> -- Sanjeet Kumar Rath, mob- +91 8777577470
Re: Listing a folder with millions of files
Thanks for the suggestions guys. The pre-filtered list is possibly one I can use. Regards, Jeremy > On 9 Mar 2020, at 20:16, Shawn Weeks wrote: > When I’ve had to do this I just skip trying to use ListFile and instead create a text file containing a list of all the files that can be used with the SplitFile and FetchFile processors to pull things in in batches. Even with filtering ListFile will iterate through a lot of files. Thanks From: Edward Armes Reply-To: "users@nifi.apache.org" Date: Monday, March 9, 2020 at 4:43 AM To: "users@nifi.apache.org" Subject: Re: Listing a folder with millions of files Hi Jeremy, In this case I don't think there is an easy answer here. You may have some luck with adjusting the max runtime of the processor but without checking the the processors implementation I couldn't know for certain if that would have any effect. Edward On Mon, 9 Mar 2020, 06:34 Jeremy Pemberton-Pigott, wrote: Hi, I need to list a sub-set (few 100,000) of files in a folder with millions of files (to do some historical processing). What's the best way I can do that? ListFiles is taking way too long and seems to try to dump the entire list to the flow when I test it on a smaller folder list. It would be good if the listing emitted files in smaller chunks so that the flow can start working on them. Regards, Jeremy
GRPC processors missing after upgrade to 1.11.3
Hey, I recently upgraded from 1.11.1 to 1.11.3 and found that the GRPC processors (ListenGRPC and InvokeGRPC) had seemingly been removed from the packaged release. But it seems that the source code in the Nifi repo for the “nifi-grpc-bundle” is still there and has not been removed. I have a couple of different versions in use and have found the bundles are missing in some but not others: - 1.11.0: GRPC nar is missing - 1.11.1: GRPC nar is included - 1.11.3: GRPC nar is missing My suspicion is the package was accidentally not included in the packaged release, is that plausible? This reminded me of another processor (ConvertJSONToAvro) which during the 1.9.x to 1.10.0 upgrade was removed without warning. At the time I put it down to the Nifi community preferring the use of “ConvertRecord” instead. Again, I found that the source code remained in the repo and was referenced in the “nifi-nar-bundles/pom.xml”. These links add to the confusion. While nifi-kite-nar seems to have been removed after 1.9.x, nifi-grpc-nar reports to not be included/referenced since 1.4, even while I have been using the processors through to 1.11.1. https://mvnrepository.com/artifact/org.apache.nifi/nifi-kite-nar https://mvnrepository.com/artifact/org.apache.nifi/nifi-grpc-nar There have not been any commits in the last few months affecting these bundles aside from release preparation commits. I have been able to copy the 1.11.1 version and run that on 1.11.3 for now, but I am hoping it can be included again in the next release. Thanks in advance, -- Dirk Arends
Re: Listing a folder with millions of files
When I’ve had to do this I just skip trying to use ListFile and instead create a text file containing a list of all the files that can be used with the SplitFile and FetchFile processors to pull things in in batches. Even with filtering ListFile will iterate through a lot of files. Thanks From: Edward Armes Reply-To: "users@nifi.apache.org" Date: Monday, March 9, 2020 at 4:43 AM To: "users@nifi.apache.org" Subject: Re: Listing a folder with millions of files Hi Jeremy, In this case I don't think there is an easy answer here. You may have some luck with adjusting the max runtime of the processor but without checking the the processors implementation I couldn't know for certain if that would have any effect. Edward On Mon, 9 Mar 2020, 06:34 Jeremy Pemberton-Pigott, mailto:fuzzych...@gmail.com>> wrote: Hi, I need to list a sub-set (few 100,000) of files in a folder with millions of files (to do some historical processing). What's the best way I can do that? ListFiles is taking way too long and seems to try to dump the entire list to the flow when I test it on a smaller folder list. It would be good if the listing emitted files in smaller chunks so that the flow can start working on them. Regards, Jeremy
Re: Nifi installation - port 8080 being used even nifi installed with a different port
You’ll need to add this line to your bootstrap.conf file to disable the Zookeeper Admin Server. We’ve done this for you in NiFi 1.11.x java.arg.17=-Dzookeeper.admin.enableServer=false From: Samarendra Sahoo Reply-To: "users@nifi.apache.org" Date: Friday, March 6, 2020 at 11:17 PM To: "users@nifi.apache.org" , "d...@nifi.apache.org" Subject: Nifi installation - port 8080 being used even nifi installed with a different port We have installed Nfi 1.10.0 on RHEL7.2 to run on port 9778. However, we are seeing that its using port 8080 as well as per the below log message. Could you pls suggest how to make all of the nifi services run on port 9778? o.a.z.server.admin.JettyAdminServer Started AdminServer on address 0.0.0.0, port 8080 Thanks Sam
Re: Listing a folder with millions of files
Hi Jeremy, In this case I don't think there is an easy answer here. You may have some luck with adjusting the max runtime of the processor but without checking the the processors implementation I couldn't know for certain if that would have any effect. Edward On Mon, 9 Mar 2020, 06:34 Jeremy Pemberton-Pigott, wrote: > Hi, > > I need to list a sub-set (few 100,000) of files in a folder with millions > of files (to do some historical processing). What's the best way I can do > that? ListFiles is taking way too long and seems to try to dump the entire > list to the flow when I test it on a smaller folder list. It would be good > if the listing emitted files in smaller chunks so that the flow can start > working on them. > > Regards, > > Jeremy >
NiFi Registry tarball modes
Moin, The jars inside of nifi-registry-0.5.0-bin.tar.gz are owned by bbende:staff with mode 0660. In a future version of the tarball, would you mind chmodding the files in lib/ and subdirectories to 0644, so I don't have to? Thanks Johannes signature.asc Description: OpenPGP digital signature
Re: Exception is showing in nifi UI users page
Hi team, I have the AD group name from the authoriser.xml file i want to delete the corresponding Unique group dientifier value from authorisation.xml file. Could you please help me to find out the file where we are keeping the relation map between AD group name and corresponding Unique group idetifier value. Thanks a lot, Sanjeet On Fri, 6 Mar 2020, 8:07 pm sanjeet rath, wrote: > Yes Mat, > > But i cant upgrading nifi till next 2 months. > As i am not able to add user so its a blocker. > > Thats why asking u how the AD group name from the authoriser file is > mapped to group identifier of authorization.xml file. > > Any place where we are storing this maping? > > So that i can delete it from authorization.xml > > Any other workaround is there. > > Thanks, > Sanjeet > > On Fri, 6 Mar 2020, 7:24 pm Matt Gilman, wrote: > >> I would recommend upgrading. If I recall correctly, there were actually a >> couple of JIRAs addressing different but similar issues dealing with users >> or groups being removed from a directory server. Even if you could identify >> and remove the problematic entries, you could possibly hit this again if >> another user or group were removed later. >> >> Thanks >> >> On Fri, Mar 6, 2020 at 6:35 AM sanjeet rath >> wrote: >> >>> Hi , >>> >>> The AD group names are present in Authoriser.xml file, >>> In authorisation.xml file the group identifier (unique id) is present >>> inside policy identifier tag. >>> So could you please help me to understand where AD group name in >>> authoriser.xml file and group identifier in authorisation.xml map together. >>> So that i can delete it in authoriser.xml file. >>> >>> Thanks, >>> Sanjeet >>> >>> >>> >>> >>> >>> >>> On Fri, Mar 6, 2020 at 12:04 AM sanjeet rath >>> wrote: >>> Thanks Mat, For this quick response. Thanks a lot , Sanjeet On Thu, 5 Mar 2020, 11:43 pm Matt Gilman, wrote: > I just responded to your StackOverflow post: > > > https://stackoverflow.com/questions/60551242/nifi-user-addition-gives-u-null-pointer-exception/60551638#60551638 > > I believe you'll need to upgrade to a version that addresses the BUG. > > Thanks! > > On Thu, Mar 5, 2020 at 1:10 PM sanjeet rath > wrote: > >> Hi Team, >> >> I am using nifi cluster from 1 month & i am able to add new user >> policies everything.its a LDAP based user addition. >> but suddenly from last 2 days , in nifi user addition page(after >> clicking on users in nifi UI) i am getting Error message "An unexcepted >> error has occure.please click logs for more details". >> and in nifif-user.log i found the bellow log. >> >> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has >> occurred: java.lang.NullPointerException. Returning Internal Server Error >> response. java.lang.NullPointerException: null at >> org.apache.nifi.web.dao.impl.StandardPolicyBasedAuthorizerDAO.lambda$null$2(StandardPolicyBasedAuthorizerDAO.java:285) >> at >> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) >> at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553) >> at >> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at >> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) >> at >> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) >> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) >> at >> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) >> >> I am not able to figuring out where should i start looking out. >> Could please someone help me to look a starting point where can check >> and what need to be checked. >> >> Thanks, >> Sanjeet >> >> >> -- >> Sanjeet Kumar Rath, >> mob- +91 8777577470 >> >> >>> >>> -- >>> Sanjeet Kumar Rath, >>> mob- +91 8777577470 >>> >>>
