What Mark said. This is the root of the other exceptions:
> io.IOException: Too many open files
↳ causes: io.netty.channel.ChannelException: failed to open a new selector
↳ causes: java.lang.IllegalStateException: failed to create a child
event loop: {}
java.lang.IllegalStateException: failed to create a child event loop
One thing I'll add is that /etc/security/limits.conf is ignored on
systemd distributions AFAIK. Not a sysadmin, but these instructions
seem very familiar from when my team fought systemd on this issue:
https://unix.stackexchange.com/a/467789
If you're using RHEL or a derivative, you'll want that starting point
otherwise you'll find none of your ulimit changes will stick.
On Wed, Jan 5, 2022 at 9:08 AM Mark Bean <mark.o.b...@gmail.com> wrote:
>
> You may want to check out the Configuration Best Practices section of the
> Admin Guide. Specifically, it mentions the following which will help with
> the "Too many open files" issue.
>
> Maximum File Handles
>
> NiFi will at any one time potentially have a very large number of file
> handles open. Increase the limits by editing */etc/security/limits.conf* to
> add something like
>
> * hard nofile 50000
> * soft nofile 50000
>
>
> On Wed, Jan 5, 2022 at 8:02 AM Martin Fong <martin.f...@toronto.ca> wrote:
>
> > We are trying to upgrade from NiFi 1.9.0 to 1.15.2 and have followed
> > modifying the nifi.properties and not copying over to attain the new
> > features + all the required users.xml, truststore.jks copied over. We
> > haven’t got to a point to really use any new features yet but whenever we
> > are trying to bring NiFi, we get the following errors:
> >
> > 2022-01-04 14:18:31,695 ERROR [Timer-Driven Process Thread-7]
> > o.a.n.c.s.StandardControllerServiceNode
> > StandardControllerServiceNode[service=DistributedMapCacheClientService[id=09a923c2-97fa-37bb-0f54-6cda853d7544],
> > name=DistributedMapCacheClientService, active=true] Failed to invoke
> > @OnEnabled method due to java.
> > io.IOException: Too many open files
> > ↳ causes: io.netty.channel.ChannelException: failed to open a new selector
> > ↳ causes: java.lang.IllegalStateException: failed to create a child event
> > loop: {}
> > java.lang.IllegalStateException: failed to create a child event loop
> > at
> > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
> > at
> > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
> > at
> > io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:52)
> > at
> > io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:97)
> > at
> > io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:92)
> > at
> > io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:73)
> > at
> > io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:53)
> > at
> > io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:45)
> >
> > 2022-01-04 14:18:31,698 ERROR [Timer-Driven Process Thread-7]
> > o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnEnabled method
> > of
> > DistributedMapCacheClientService[id=09a923c2-97fa-37bb-0f54-6cda853d7544]
> > due to java.lang.IllegalStateException: failed to create a child event loop
> > 2022-01-04 14:18:31,698 INFO [main] o.a.n.c.s.StandardProcessScheduler
> > Enabling
> > StandardControllerServiceNode[service=AvroSchemaRegistry[id=00f0fc9e-63b4-37d2-65b4-4b5688ac2673],
> > name=CSU_SCHEMA_REGISTRY, active=false]
> > 2022-01-04 14:18:31,698 ERROR [Timer-Driven Process Thread-7]
> > o.a.n.c.s.StandardControllerServiceNode
> > StandardControllerServiceNode[service=DistributedMapCacheClientService[id=09a923c2-97fa-37bb-0f54-6cda853d7544],
> > name=DistributedMapCacheClientService, active=true] Failed to invoke
> > @OnDisabled method due to java.lang.NullPointerException: {}
> > java.lang.NullPointerException: null
> > at
> > org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.onDisabled(DistributedMapCacheClientService.java:122)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
> > at
> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
> > at
> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
> > at
> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
> > at
> > org.apache.nifi.controller.service.StandardControllerServiceNode.invokeDisable(StandardControllerServiceNode.java:650)
> > at
> > org.apache.nifi.controller.service.StandardControllerServiceNode.access$500(StandardControllerServiceNode.java:85)
> > at
> > org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:574)
> > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2022-01-04 14:18:31,699 ERROR [Timer-Driven Process Thread-7]
> > o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnDisabled method
> > of
> > DistributedMapCacheClientService[id=09a923c2-97fa-37bb-0f54-6cda853d7544]
> > due to java.lang.NullPointerException
> >
> >
> > 2022-01-04 14:18:57,522 INFO [main]
> > o.a.n.wali.SequentialAccessWriteAheadLog Recovering records from
> > Write-Ahead Log at /application/repository/flowfile_repository
> > 2022-01-04 14:18:57,530 INFO [main]
> > o.apache.nifi.controller.FlowController Initiated graceful shutdown of flow
> > controller...waiting up to 10 seconds
> > 2022-01-04 14:19:01,536 INFO [Cleanup Archive for default]
> > o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0
> > bytes) from archive
> > 2022-01-04 14:19:03,262 WARN [main]
> > o.apache.nifi.controller.FlowController Controller hasn't terminated
> > properly. There exists an uninterruptable thread that will take an
> > indeterminate amount of time to stop. Might need to kill the program
> > manually.
> > 2022-01-04 14:19:03,274 WARN [main] org.eclipse.jetty.webapp.WebAppContext
> > Failed startup of context o.e.j.w.WebAppContext@3fe98084
> > {nifi-api,/nifi-api,file:///application/nifi-1.15.2/work/jetty/nifi-web-api-1.15.2.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.15.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.15.2.war}
> > org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
> > at
> > org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:83)
> > at
> > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1067)
> > at
> > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
> > at
> > org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:996)
> >
> > 2022-01-04 14:19:03,548 WARN [main] org.eclipse.jetty.webapp.WebAppContext
> > Failed startup of context o.e.j.w.WebAppContext@5f33e6d
> > {nifi-content-viewer,/nifi-content-viewer,file:///application/nifi-1.15.2/work/jetty/nifi-web-content-viewer-1.15.2.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.15.2.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-content-viewer-1.15.2.war}
> > javax.servlet.UnavailableException: Class loading error for holder
> > ContentViewerController==org.apache.nifi.web.ContentViewerController@d70bc1c7
> > {jsp=null,order=-1,inst=false,async=false,src=DESCRIPTOR:file:///application/nifi-1.15.2/work/jetty/nifi-web-content-viewer-1.15.2.war/webapp/WEB-INF/web.xml,STARTING}
> > at org.eclipse.jetty.servlet.BaseHolder.doStart(BaseHolder.java:96)
> >
> > Eventually it will shut down. For testing purposes, I have copied the
> > nifi.properties from 1.9.0 and was hoping NiFi would come up and still the
> > same issue.
> >
> > I have used the same steps migrating from 1.9.0 to 1.12.1 and it was fine.
> >
> > I am new joining to the NiFi support and if there is a forum that I need
> > to join to get support, please let me know the proper channel.
> >
> > Please advise,
> > Martin Fong
> > Enterprise Technical Support Specialist, Infrastructure & Platform (IAG)
> > Technology Services Division, Technology Infrastructure Services
> > City of Toronto
> > 703 Don Mills Road, 2nd Floor
> > Toronto, ON
> > M3C 3N3
> > Tel: 416-397-7565
> > e-mail: martin.f...@toronto.ca<mailto:martin.f...@toronto.ca>
> >
> > This e-mail message is confidential and subject to copyright. Any
> > unauthorized use or disclosure is prohibited. If you have received this
> > email and are not the intended recipient, please advise and delete it.
> > Thank you.
> >
> >
