Hi Jonathan, I found following counts in /proc/$PID/fd folder.
Socket: 7807 Pipe: 76 anon_inode:[eventpoll]: 38 /usr/lib/jvm/jdk1.8.0_92/ : 5 /appl/myapplicationfolder : 6 Thanks & Regards, Krishan Babbar ============================================================================================================================ Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra. =========================================================================================================================== -----Original Message----- From: Jonathan Valliere <[email protected]> Sent: Wednesday, December 5, 2018 4:34 PM To: [email protected] Cc: Atul Kandhari <[email protected]> Subject: Re: Getting "Too many open files" warnings11 Can you find out the open file by type instead of just the number? How many are storage files, how many are sockets? On Wed, Dec 5, 2018 at 2:10 AM Stefan Magnus Landrø <[email protected]> wrote: > Perform a heap dump to find out what’s going on > > Sendt fra min iPhone > > > 5. des. 2018 kl. 07:36 skrev Krishan Babbar > ><[email protected] > >: > > > > Any idea why file count is increasing day by day? Can we close any > socket/file handle manually? > > > > Date > > > > Time > > > > Open files count > > > > 2-Dec > > > > 16:08 PM > > > > 7047 > > > > 3-Dec > > > > 8:29 AM > > > > 7209 > > > > 3-Dec > > > > 14:45 PM > > > > 7289 > > > > 3-Dec > > > > 16:00 PM > > > > 7308 > > > > 3-Dec > > > > 16:11 PM > > > > 7310 > > > > 3-Dec > > > > 18:01 PM > > > > 7330 > > > > 3-Dec > > > > 20:00 PM > > > > 7358 > > > > 3-Dec > > > > 22:00 PM > > > > 7394 > > > > 4-Dec > > > > 8:25 AM > > > > 7501 > > > > 4-Dec > > > > 11:09 AM > > > > 7543 > > > > 4-Dec > > > > 12:00 PM > > > > 7551 > > > > 4-Dec > > > > 14:00 PM > > > > 7594 > > > > 4-Dec > > > > 16:00 PM > > > > 7614 > > > > 4-Dec > > > > 18:00 PM > > > > 7635 > > > > 4-Dec > > > > 20:00 PM > > > > 7668 > > > > 4-Dec > > > > 22:00 PM > > > > 7689 > > > > > > > > Thanks & Regards, > > Krishan Babbar > > > > > ====================================================================== > ====================================================== > > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html externally > http://tim.techmahindra.com/tim/disclaimer.html internally within > TechMahindra. > > > ====================================================================== > ===================================================== > > > > From: Krishan Babbar > > Sent: Monday, December 3, 2018 11:27 AM > > To: [email protected] > > Cc: Atul Kandhari <[email protected]> > > Subject: RE: Getting "Too many open files" warnings11 > > > > > > Thanks Jonathan, > > > > > > > > Ok, but I did not see any message like "acceptor was sleeping". > > > > > > > > I noticed that my open files count for my Java application is keep > increasing day by day as shown below. Why is it not decreasing again? > > > > What configuration am I missing? What can be the reason? > > > > > > Date > > > > Time > > > > Open file count > > > > 30-Nov > > > > 20:00 PM > > > > 2273 > > > > 30-Nov > > > > 22:00 PM > > > > 2544 > > > > 1-Dec > > > > 12:00 AM > > > > 2790 > > > > 1-Dec > > > > 2:00 AM > > > > 2987 > > > > 1-Dec > > > > 4:00 AM > > > > 3283 > > > > 1-Dec > > > > 12:00 PM > > > > 4828 > > > > 1-Dec > > > > 14:00 PM > > > > 5200 > > > > 1-Dec > > > > 16:00 PM > > > > 5635 > > > > 2-Dec > > > > 10:17 AM > > > > 6992 > > > > 2-Dec > > > > 12:00 PM > > > > 7008 > > > > 2-Dec > > > > 14:02 PM > > > > 7024 > > > > 2-Dec > > > > 16:08 PM > > > > 7047 > > > > > > > > > > > > > > In my RequestHandler, I have following settings, means setting > > session > timeout to 1800 seconds (30 Mins) > > @Override > > public void messageReceived(IoSession session, Object message) > throws Exception { > > if (session != null) { > > this.session = session; > > this.session.getConfig().setBothIdleTime(1800); > > LOGGER.info("Session Idle time in sec: " + > this.session.getConfig().getBothIdleTime()); > > } > > > > this.message = message; > > > > } > > > > > > > > Yes, I can simulate multiple devices with lowering nlimit and test. > > What > would be the next? > > > > > > > > Thanks & Regards, > > > > Krishan Babbar > > > > > > > > > ====================================================================== > ====================================================== > > > > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html externally > http://tim.techmahindra.com/tim/disclaimer.html internally within > TechMahindra. > > > > > ====================================================================== > ===================================================== > > > > > > > > -----Original Message----- > > From: Jonathan Valliere <[email protected]<mailto: > [email protected]>> > > Sent: Friday, November 30, 2018 7:20 PM > > To: [email protected]<mailto:[email protected]> > > Cc: Atul Kandhari <[email protected]<mailto: > [email protected]>> > > Subject: Re: Getting "Too many open files" warnings11 > > > > > > > > Mina is patched to prevent that exception from occurring in a cycle. > > That > > > > was a long time ago, your version is patched. You probably noticed a > > > > message saying the acceptor was sleeping. > > > > > > > > Do you have any kind of testing utility to create fake users for > > your > application? I would recommend lowering the nlimit and trying to > reproduce > > > > entirely artificially. There shouldn’t be any correlation to > > > > > > > > On Fri, Nov 30, 2018 at 8:39 AM Krishan Babbar < > [email protected]<mailto:[email protected]>> > > > > wrote: > > > > > > > >> Hi Jonathan, > > > >> > > > >> Version - 2.0.16 > > > >> Java Application process was using around 1% of CPU and 3.4% of Memory. > > > >> > > > >> After mitigate, were you able to provide the solution? > > > >> I mean what would be the solution if you know? > > > >> > > > >> Thanks & Regards, > > > >> Krishan Babbar > > > >> > > > >> -----Original Message----- > > > >> From: Jonathan Valliere <[email protected]<mailto: > [email protected]>> > > > >> Sent: Friday, November 30, 2018 6:56 PM > > > >> To: [email protected]<mailto:[email protected]> > > > >> Cc: Atul Kandhari <[email protected]<mailto: > [email protected]>> > > > >> Subject: Re: Getting "Too many open files" warnings11 > > > >> > > > >> What version of Mina are you using? We added code to mitigate this > > > >> specific exemption a while back. > > > >> > > > >> Do you know how much cpu Java was consuming at that point in time? > > > >> > > > >> On Fri, Nov 30, 2018 at 8:24 AM Krishan Babbar < > > > >> [email protected]<mailto:[email protected]>> > > > >> wrote: > > > >> > > > >>> Thanks Stefan, > > > >>> > > > >>> It would be great if you could answers other questions too. > > > >>> -What do you think, why application stops receiving packets from > devices? > > > >>> Is it only due to exception "Too many open files" or it can be due > > > >>> to any other reasons as well? Please guide. > > > >>> -Is my MINA configuration fine shown in below Java code? Please > > > >>> suggest if I need to fine tune it more. > > > >>> > > > >>> Thanks & Regards, > > > >>> Krishan Babbar > > > >>> > > > >>> > > > >>> > > > >>> ================================================================== > >>> == > > > >>> == ====================================================== > > > >>> Disclaimer: This message and the information contained herein is > > > >>> proprietary and confidential and subject to the Tech Mahindra > >>> policy > > > >>> statement, you may review the policy at > > > >>> http://www.techmahindra.com/Disclaimer.html externally > > > >>> http://tim.techmahindra.com/tim/disclaimer.html internally within > > > >>> TechMahindra. > > > >>> > > > >>> ================================================================== > >>> == > > > >>> == ===================================================== > > > >>> > > > >>> -----Original Message----- > > > >>> From: Stefan Magnus Landrø <[email protected]<mailto: > [email protected]>> > > > >>> Sent: Friday, November 30, 2018 5:59 PM > > > >>> To: [email protected]<mailto:[email protected]> > > > >>> Cc: Atul Kandhari <[email protected]<mailto: > [email protected]>> > > > >>> Subject: Re: Getting "Too many open files" warnings11 > > > >>> > > > >>> > > > >>> https://unix.stackexchange.com/questions/108603/do-changes-in-etc- > >>> se > > > >>> cu > > > >>> rity-limits-conf-require-a-reboot > > > >>> > > > >>> > > > >>> Sendt fra min iPhone > > > >>> > > > >>>> 30. nov. 2018 kl. 11:17 skrev Krishan Babbar > > > >>>> <[email protected] > > > >>>> : > > > >>>> > > > >>>> Dear All, > > > >>>> > > > >>>> I am working on an IoT project. I have created a Java application > > > >>>> using > > > >>> MINA library. All the devices are connecting to this Java > > > >>> application using TCP protocol. I am getting around 3000 packets > >>> per > > > >>> hour from about 50 devices as of now and it will increase soon. > > > >>>> > > > >>>> I am facing one issue on production environment. After running > >>>> say > > > >>>> 1-2 > > > >>> days, my application stops receiving packets from all the devices > > > >>> but application process keep running. To make it working again, I > > > >>> need to restart my Java application. I noticed following warnings > >>> in > logs. > > > >>>> Warnings > > > >>>> [NioSocketAcceptor-3] WARN > > > >>>> org.apache.mina.util.DefaultExceptionMonitor > > > >>> 30/11/2018 05:25:40 - Unexpected exception. > > > >>>> java.io.IOException: Too many open files > > > >>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native > >>>> Method) > > > >>>> at sun.nio.ch > > > >>> .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) > > > >>>> at sun.nio.ch > > > >>> .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > > > >>>> at > > > >>> org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioS > >>> oc > > > >>> ke > > > >>> tAcceptor.java:194) > > > >>>> at > > > >>> org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioS > >>> oc > > > >>> ke > > > >>> tAcceptor.java:51) > > > >>>> at > > > >>> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.pr > >>> oc > > > >>> es > > > >>> sHandles(AbstractPollingIoAcceptor.java:544) > > > >>>> at > > > >>> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.ru > >>> n( > > > >>> Ab > > > >>> stractPollingIoAcceptor.java:484) > > > >>>> at > > > >>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunn > >>> ab > > > >>> le > > > >>> .java:64) > > > >>>> at > > > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut > >>> or > > > >>> .j > > > >>> ava:1142) > > > >>>> at > > > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > >>> java:617) > > > >>>> at java.lang.Thread.run(Thread.java:745) > > > >>>> > > > >>>> Due to above warnings in logs, yesterday, after doing some > >>>> Google, > > > >>>> I > > > >>> changed file limits in "/etc/security/limits.conf" file to 100000 > >>> as > > > >>> shown below, but I did not restart my Java application. > > > >>>> * hard nofile 100000 > > > >>>> * soft nofile 100000 > > > >>>> > > > >>>> Today. I got the same warnings again and I was not getting any > > > >>>> packet > > > >>> from any of the devices. So I restarted my Java application and > > > >>> devices started sending packets again. > > > >>>> Now, my questions are > > > >>>> > > > >>>> - After application stopped receiving packets, I restarted my > > > >>> Java application today. Will it consider new file limits i.e. > >>> 100000 > > > >>> and not give Exceptions again? Or do I need to reboot machine as well? > > > >>>> > > > >>>> - What do you think, why application stops receiving packets > > > >>> from devices? Is it only due to exception "Too many open files" or > > > >>> it can be due to any other reasons as well? Please guide. > > > >>>> > > > >>>> - Is my MINA configuration fine shown in below Java code? > > > >>> Please suggest if I need to fine tune it more. > > > >>>> > > > >>>> > > > >>>> Below is given our code for MINA configuration. > > > >>>> import java.io.IOException; > > > >>>> import java.net.InetSocketAddress; import > >>>> org.apache.log4j.Logger; > > > >>>> import org.apache.mina.core.service.AbstractIoAcceptor; > > > >>>> import org.apache.mina.core.session.IdleStatus; > > > >>>> import org.apache.mina.filter.logging.LoggingFilter; > > > >>>> import org.apache.mina.transport.socket.nio.NioDatagramAcceptor; > > > >>>> > > > >>>> import com.techm.tnt.protocol.gateway.util.ServerConfigLoader; > > > >>>> > > > >>>> /** > > > >>>> * The abstract class for tcp/ip and udp listeners. > > > >>>> */ > > > >>>> public abstract class AbstractListener { > > > >>>> > > > >>>> /** The logger instance.*/ > > > >>>> private static Logger LOGGER = > > > >>>> Logger.getLogger(AbstractListener.class); > > > >>>> > > > >>>> /** The buffer size constant. */ > > > >>>> protected static final int BUFFER_SIZE = 2048; > > > >>>> > > > >>>> /** > > > >>>> * The method to initialize udp and tcp/ip listeners. > > > >>>> * > > > >>>> * @param portKey > > > >>>> * the port of the udp or tcp/ip listener. > > > >>>> * @throws IOException > > > >>>> * the io exception to be thrown in case of > > > >>> error. > > > >>>> */ > > > >>>> public void init(String portKey) throws IOException > >>>> { > > > >>>> LOGGER.info("Entering init()"); > > > >>>> AbstractIoAcceptor acceptor = > > > >>> initInternal(); > > > >>>> acceptor.setHandler(new > > > >>>> RequestHandler()); > > > >>>> > > > >>> acceptor.getFilterChain().addLast("logger", new LoggingFilter()); > > > >>>> > > > >>> acceptor.getSessionConfig().setReadBufferSize(BUFFER_SIZE); > > > >>>> > > > >>> acceptor.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE, 10); > > > >>>> if (acceptor instanceof > > > >>> NioDatagramAcceptor) { > > > >>>> > > > >>>> ((NioDatagramAcceptor) > > > >>> acceptor).getSessionConfig().setReuseAddress(true); > > > >>>> } > > > >>>> acceptor.bind(new > > > >>> InetSocketAddress(Integer.parseInt(ServerConfigLoader.getInstance(). > > > >>> ge > > > >>> tProperty(portKey)))); > > > >>>> LOGGER.info("Exiting init()"); > > > >>>> } > > > >>>> > > > >>>> /** > > > >>>> * The abstract method to be over ridden by > > > >>>> appropriate > > > >>> listener (udp / > > > >>>> * tcp/ip) for invoking appropriate listener. > > > >>>> * > > > >>>> * @return the appropriate (udp/ tcp/ip) io acceptor. > > > >>>> */ > > > >>>> public abstract AbstractIoAcceptor initInternal(); > > > >>>> } > > > >>>> > > > >>>> Appreciate all tour help. > > > >>>> > > > >>>> Thanks & Regards, > > > >>>> Krishan Babbar > > > >>>> > > > >>>> > > > >>>> > > > >>>> ================================================================= > >>>> = > > > >>>> == == ====================================================== > > > >>>> Disclaimer: This message and the information contained herein is > > > >>> proprietary and confidential and subject to the Tech Mahindra > >>> policy > > > >>> statement, you may review the policy at > > > >>> http://www.techmahindra.com/Disclaimer.html externally > > > >>> http://tim.techmahindra.com/tim/disclaimer.html internally within > > > >>> TechMahindra. > > > >>>> ================================================================= > >>>> = > > > >>>> == == ===================================================== > > > >>>> > > > >>>> ================================================================= > >>>> = > > > >>>> == == ====================================================== > > > >>>> > > > >>>> Disclaimer: This message and the information contained herein is > > > >>> proprietary and confidential and subject to the Tech Mahindra > >>> policy > > > >>> statement, you may review the policy at > > > >>> http://www.techmahindra.com/Disclaimer.html < > > > >>> http://www.techmahindra.com/Disclaimer.html> externally > > > >>> http://tim.techmahindra.com/tim/disclaimer.html < > > > >>> http://tim.techmahindra.com/tim/disclaimer.html> internally within > > > >>> TechMahindra. > > > >>>> > > > >>>> ================================================================= > >>>> = > > > >>>> == == ====================================================== > > > >>> > > > >> -- > > > >> > > > >> CONFIDENTIALITY NOTICE: The contents of this email message and any > > > >> attachments are intended solely for the addressee(s) and may > >> contain > > > >> confidential and/or privileged information and may be legally > > > >> protected from disclosure. > > > >> > > > >> =================================================================== > >> === > > > >> ====================================================== > > > >> > > > >> Disclaimer: This message and the information contained herein is > > > >> proprietary and confidential and subject to the Tech Mahindra > >> policy > > > >> statement, you may review the policy at > > > >> http://www.techmahindra.com/Disclaimer.html < > > > >> http://www.techmahindra.com/Disclaimer.html> externally > > > >> http://tim.techmahindra.com/tim/disclaimer.html < > > > >> http://tim.techmahindra.com/tim/disclaimer.html> internally within > > > >> TechMahindra. > > > >> > > > >> > > > >> =================================================================== > >> === > > > >> ====================================================== > > > >> > > > > -- > > > > > > > > CONFIDENTIALITY NOTICE: The contents of this email message and any > attachments are intended solely for the addressee(s) and may contain > confidential and/or privileged information and may be legally > protected from disclosure. > > > ====================================================================== > ====================================================== > > > > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html < > http://www.techmahindra.com/Disclaimer.html> externally > http://tim.techmahindra.com/tim/disclaimer.html < > http://tim.techmahindra.com/tim/disclaimer.html> internally within > TechMahindra. > > > > > ====================================================================== > ====================================================== > ============================================================================================================================ Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html <http://www.techmahindra.com/Disclaimer.html> externally http://tim.techmahindra.com/tim/disclaimer.html <http://tim.techmahindra.com/tim/disclaimer.html> internally within TechMahindra. ============================================================================================================================
