I was thinking that with 16000 processors, you would need a large amount of heap space allocated to the JVM. A small heap would slow NiFi's ability to do its work, because the JVM garbage collector would be consuming more CPU time than what is healthy. A 64 GB heap size, allocated in the NiFi boostrap.conf, can work. Hopefully the system you are running this NiFi has adequate CPU and RAM resources.
-- Mike On Fri, Dec 29, 2017 at 9:28 AM, Mike Thomsen <mikerthom...@gmail.com> wrote: > How many of these 1600 processor groups represent distinct functionality > that cannot be genericized? > > Thanks, > > Mike > > On Thu, Dec 28, 2017 at 8:54 PM, <l...@china-inv.cn> wrote: > > > Thanks for your help. > > > > Actually we are using Kylo (https://kylo.io/) which uses NiFi to do the > > work. > > > > There are about 10 processors in each group, so there will be 16000 > > processors in total. > > > > I understand that the processor should be validate before it starts, but > > could this validation be defered until the processor (or the group it > > belongs to) is to start for the first time > > instead of doing this when NiFi service starts up? > > > > > > > > 发件人: > > Joe Witt <joe.w...@gmail.com> > > 收件人: > > dev@nifi.apache.org > > 日期: > > 2017/12/29 09:20 > > 主题: > > Re: 答复: Re: NiFi starts very slowly > > > > > > > > hello > > > > the total number of processors is probably a factor. how many are there? > > can you please list how many processors of each type there are? Any > > custom > > processors? > > > > creating 1600 disjoint flows is probably going against the grain of nifi. > > cant the flows be designed to operate as a directed graph amd leverage > > attributes? > > > > keep in mind to start each processor we call validate. this could be the > > cause of the slowness. > > > > help us collect the above info and we can give pointers. if you could > > share your flow.xml.gz that would be great. > > > > thanks > > joe > > > > On Dec 28, 2017 8:13 PM, <l...@china-inv.cn> wrote: > > > > > Thanks for your reply. > > > > > > We are not using a cluster, our NiFi 1.3 runs in the single node mode. > > > > > > > > > > > > 发件人: > > > "Mohammed Nadeem" <nadeemm...@gmail.com> > > > 收件人: > > > dev@nifi.apache.org > > > 日期: > > > 2017/12/28 20:11 > > > 主题: > > > Re: NiFi starts very slowly > > > > > > > > > > > > Hi Luby, > > > > > > If you are using a cluster based setup then Nifi is unstable due to > > nodes > > > getting disconnected by the cluster cordinator. Cluster coordinator > > > disconnects nodes from the cluster when the nodes take a little longer > > to > > > respond ( beyond timeout) . The cluster coordinator timeout based on > the > > > settings in the nifi.properties file. > > > > > > We need to adjust the following properties > > > > > > *nifi.cluster.node.connection.timeout * = 5 sec ( by default ) > > > *nifi.cluster.node.read.timeout *= 5 sec ( by default) > > > > > > By Increasing the timeout ,from 5 sec to higher value would probably > > solve > > > a > > > problem > > > > > > > > > Thanks, > > > Nadeem > > > > > > > > > > > > -- > > > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ > > > > > > > > > > > > > > > > > > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对 > > 外 > > > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发 > > 件 > > > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。 > > > > > > > > > This email message may contain confidential and/or privileged > > information. > > > If you are not the intended recipient, please do not read, save, > > forward, > > > disclose or copy the contents of this email or open any file attached > to > > > this email. We will be grateful if you could advise the sender > > immediately > > > by replying this email, and delete this email and any attachment or > > links > > > to this email completely and immediately from your computer system. > > > > > > > > > > > > > > > > > > > > > > > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对外 > > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发件 > > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。 > > > > > > This email message may contain confidential and/or privileged > information. > > If you are not the intended recipient, please do not read, save, forward, > > disclose or copy the contents of this email or open any file attached to > > this email. We will be grateful if you could advise the sender > immediately > > by replying this email, and delete this email and any attachment or links > > to this email completely and immediately from your computer system. > > > > > > > > >
