By Hai Luo, I haven't personally attempted to use NiFi with Hadoop in the way you described. If your Hadoop clusters are not secured I don't think you'll have issues with UGI. I wouldn't recommend running NiFi and Hadoop clusters unsecured, however.
On Tue, Nov 12, 2019 at 8:40 PM abellnotring <abellnotr...@sina.com> wrote: > Hi,jeff, > There is no Kerberos authentication in my HADOOP clusters , but I > find UGI is initialized with an ExtendConfiguration(extend from hadoop > Configruation) when those processor's instance was first scheduled .I would > use nifi to connect different HADOOP cluster, will it run into any > issues?(I’m running tests for this) > > > Thanks > By Hai Luo > On 11/13/2019 07:01，Jeff<jtsw...@gmail.com> <jtsw...@gmail.com> wrote： > > If you remove the @RequiresInstanceClassloading, the UserGroupInformation > class from Hadoop (hadoop-common, if I remember correctly) will be shared > across all instances that come from a particular NAR (such as PutHDFS, > ListHDFS, FetchHDFS, etc, from nifi-hadoop-nar-x.y.z.nar). If you are > using Kerberos in those processors and configured different principals > across the various processors, you could run into issues when the > processors attempt to acquire new TGTs, most likely the first time a > relogin is attempted. UGI has some static state and > @RequiresInstanceClassloading makes sure each instance of a processor with > that annotation has its own classloader to keep that kind of state from > being shared across instances. > > On Mon, Nov 11, 2019 at 9:41 PM abellnotring <abellnotr...@sina.com> > wrote: > >> Hi,Peter & All >> I’m using kylo to manage the nifi flow(called feed in kylo),and >> there are 4200 instances(600+ instances extended from >> AbstractHadoopProcessor) in my nifi canvas,The NIFI Non-Heap Memory has >> increased more than 6GB after some days running ,which is extremely >> abnormal . I have analyzed the class loaded into Compressed Class Space >> ,and found most of the CCS was used by classes related by >> AbstractHadoopProcessor. >> So I think removing RequiresInstanceClassLoading from >> AbstractHadoopProcessor processor may be a Solution for reducing the CCS >> used. >> Do you have any ideas for this ? >> >> >> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=abellnotring&uid=abellnotring%40sina.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22abellnotring%40sina.com%22%5D> >> >> >> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=abellnotring&uid=abellnotring%40sina.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22abellnotring%40sina.com%22%5D> >> >> >> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=abellnotring&uid=abellnotring%40sina.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22abellnotring%40sina.com%22%5D> >> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=abellnotring&uid=abellnotring%40sina.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22abellnotring%40sina.com%22%5D> >> Thanks >> >> >> By Hai Luo >> On 11/12/2019 02:17，Shawn Weeks<swe...@weeksconsulting.us> >> <swe...@weeksconsulting.us> wrote： >> >> I’m assuming your talking about the snappy problem. If you use compress >> content prior to puthdfs you can compress with Snappy as it uses the Java >> Native Snappy Lib. The HDFS processors are limited to the actual Hadoop >> Libraries so they’d have to change from Native to get around this. I’m >> pretty sure we need instance loading to handle the other issues mentioned. >> >> >> >> Thanks >> >> Shawn >> >> >> >> *From: *Joe Witt <joe.w...@gmail.com> >> *Reply-To: *"users@nifi.apache.org" <users@nifi.apache.org> >> *Date: *Monday, November 11, 2019 at 8:56 AM >> *To: *"users@nifi.apache.org" <users@nifi.apache.org> >> *Subject: *Re: Influence about removing RequiresInstanceClassLoading >> from AbstractHadoopProcessor processor >> >> >> >> Peter >> >> >> >> The most common challenge is if two isolated instances both want to use a >> native lib. No two native libs with the same name can be in the same jvm. >> We need to solve that for sure. >> >> >> >> Thanks >> >> >> >> On Mon, Nov 11, 2019 at 9:53 AM Peter Turcsanyi <turcsa...@apache.org> >> wrote: >> >> Hi Hai Luo, >> >> >> >> @RequiresInstanceClassLoading makes possible to configure separate / >> isolated "Additional Classpath Resources" settings on your HDFS processors >> (eg. S3 storage driver on one of your PutHDFS and Azure Blob on the other). >> >> >> >> Is there any specific reason / use case why you are considering to remove >> it? >> >> >> >> Regards, >> >> Peter Turcsanyi >> >> >> >> On Mon, Nov 11, 2019 at 3:30 PM abellnotring <abellnotr...@sina.com> >> wrote: >> >> Hi,all >> >> I’m considering removing the RequiresInstanceClassLoading annotation >> from class AbstractHadoopProcessor, >> >> Does anybody know the potential Influence? >> >> >> >> Thanks >> >> By Hai Luo >> >>
