Any idea why file count is increasing day by day? Can we close any socket/file handle manually?
Date Time Open files count 2-Dec 16:08 PM 7047 3-Dec 8:29 AM 7209 3-Dec 14:45 PM 7289 3-Dec 16:00 PM 7308 3-Dec 16:11 PM 7310 3-Dec 18:01 PM 7330 3-Dec 20:00 PM 7358 3-Dec 22:00 PM 7394 4-Dec 8:25 AM 7501 4-Dec 11:09 AM 7543 4-Dec 12:00 PM 7551 4-Dec 14:00 PM 7594 4-Dec 16:00 PM 7614 4-Dec 18:00 PM 7635 4-Dec 20:00 PM 7668 4-Dec 22:00 PM 7689 Thanks & Regards, Krishan Babbar ============================================================================================================================ Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra. =========================================================================================================================== From: Krishan Babbar Sent: Monday, December 3, 2018 11:27 AM To: [email protected] Cc: Atul Kandhari <[email protected]> Subject: RE: Getting "Too many open files" warnings11 Thanks Jonathan, Ok, but I did not see any message like "acceptor was sleeping". I noticed that my open files count for my Java application is keep increasing day by day as shown below. Why is it not decreasing again? What configuration am I missing? What can be the reason? Date Time Open file count 30-Nov 20:00 PM 2273 30-Nov 22:00 PM 2544 1-Dec 12:00 AM 2790 1-Dec 2:00 AM 2987 1-Dec 4:00 AM 3283 1-Dec 12:00 PM 4828 1-Dec 14:00 PM 5200 1-Dec 16:00 PM 5635 2-Dec 10:17 AM 6992 2-Dec 12:00 PM 7008 2-Dec 14:02 PM 7024 2-Dec 16:08 PM 7047 In my RequestHandler, I have following settings, means setting session timeout to 1800 seconds (30 Mins) @Override public void messageReceived(IoSession session, Object message) throws Exception { if (session != null) { this.session = session; this.session.getConfig().setBothIdleTime(1800); LOGGER.info("Session Idle time in sec: " + this.session.getConfig().getBothIdleTime()); } this.message = message; } Yes, I can simulate multiple devices with lowering nlimit and test. What would be the next? Thanks & Regards, Krishan Babbar ============================================================================================================================ Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra. =========================================================================================================================== -----Original Message----- From: Jonathan Valliere <[email protected]<mailto:[email protected]>> Sent: Friday, November 30, 2018 7:20 PM To: [email protected]<mailto:[email protected]> Cc: Atul Kandhari <[email protected]<mailto:[email protected]>> Subject: Re: Getting "Too many open files" warnings11 Mina is patched to prevent that exception from occurring in a cycle. That was a long time ago, your version is patched. You probably noticed a message saying the acceptor was sleeping. Do you have any kind of testing utility to create fake users for your application? I would recommend lowering the nlimit and trying to reproduce entirely artificially. There shouldn’t be any correlation to On Fri, Nov 30, 2018 at 8:39 AM Krishan Babbar <[email protected]<mailto:[email protected]>> wrote: > Hi Jonathan, > > Version - 2.0.16 > Java Application process was using around 1% of CPU and 3.4% of Memory. > > After mitigate, were you able to provide the solution? > I mean what would be the solution if you know? > > Thanks & Regards, > Krishan Babbar > > -----Original Message----- > From: Jonathan Valliere > <[email protected]<mailto:[email protected]>> > Sent: Friday, November 30, 2018 6:56 PM > To: [email protected]<mailto:[email protected]> > Cc: Atul Kandhari > <[email protected]<mailto:[email protected]>> > Subject: Re: Getting "Too many open files" warnings11 > > What version of Mina are you using? We added code to mitigate this > specific exemption a while back. > > Do you know how much cpu Java was consuming at that point in time? > > On Fri, Nov 30, 2018 at 8:24 AM Krishan Babbar < > [email protected]<mailto:[email protected]>> > wrote: > > > Thanks Stefan, > > > > It would be great if you could answers other questions too. > > -What do you think, why application stops receiving packets from devices? > > Is it only due to exception "Too many open files" or it can be due > > to any other reasons as well? Please guide. > > -Is my MINA configuration fine shown in below Java code? Please > > suggest if I need to fine tune it more. > > > > Thanks & Regards, > > Krishan Babbar > > > > > > > > ==================================================================== > > == ====================================================== > > Disclaimer: This message and the information contained herein is > > proprietary and confidential and subject to the Tech Mahindra policy > > statement, you may review the policy at > > http://www.techmahindra.com/Disclaimer.html externally > > http://tim.techmahindra.com/tim/disclaimer.html internally within > > TechMahindra. > > > > ==================================================================== > > == ===================================================== > > > > -----Original Message----- > > From: Stefan Magnus Landrø > > <[email protected]<mailto:[email protected]>> > > Sent: Friday, November 30, 2018 5:59 PM > > To: [email protected]<mailto:[email protected]> > > Cc: Atul Kandhari > > <[email protected]<mailto:[email protected]>> > > Subject: Re: Getting "Too many open files" warnings11 > > > > > > https://unix.stackexchange.com/questions/108603/do-changes-in-etc-se > > cu > > rity-limits-conf-require-a-reboot > > > > > > Sendt fra min iPhone > > > > > 30. nov. 2018 kl. 11:17 skrev Krishan Babbar > > ><[email protected] > > >: > > > > > > Dear All, > > > > > > I am working on an IoT project. I have created a Java application > > > using > > MINA library. All the devices are connecting to this Java > > application using TCP protocol. I am getting around 3000 packets per > > hour from about 50 devices as of now and it will increase soon. > > > > > > I am facing one issue on production environment. After running say > > > 1-2 > > days, my application stops receiving packets from all the devices > > but application process keep running. To make it working again, I > > need to restart my Java application. I noticed following warnings in logs. > > > Warnings > > > [NioSocketAcceptor-3] WARN > > > org.apache.mina.util.DefaultExceptionMonitor > > 30/11/2018 05:25:40 - Unexpected exception. > > > java.io.IOException: Too many open files > > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > > at sun.nio.ch > > .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) > > > at sun.nio.ch > > .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > > > at > > org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioSoc > > ke > > tAcceptor.java:194) > > > at > > org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioSoc > > ke > > tAcceptor.java:51) > > > at > > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.proc > > es > > sHandles(AbstractPollingIoAcceptor.java:544) > > > at > > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run( > > Ab > > stractPollingIoAcceptor.java:484) > > > at > > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnab > > le > > .java:64) > > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor > > .j > > ava:1142) > > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > java:617) > > > at java.lang.Thread.run(Thread.java:745) > > > > > > Due to above warnings in logs, yesterday, after doing some Google, > > > I > > changed file limits in "/etc/security/limits.conf" file to 100000 as > > shown below, but I did not restart my Java application. > > > * hard nofile 100000 > > > * soft nofile 100000 > > > > > > Today. I got the same warnings again and I was not getting any > > > packet > > from any of the devices. So I restarted my Java application and > > devices started sending packets again. > > > Now, my questions are > > > > > > - After application stopped receiving packets, I restarted my > > Java application today. Will it consider new file limits i.e. 100000 > > and not give Exceptions again? Or do I need to reboot machine as well? > > > > > > - What do you think, why application stops receiving packets > > from devices? Is it only due to exception "Too many open files" or > > it can be due to any other reasons as well? Please guide. > > > > > > - Is my MINA configuration fine shown in below Java code? > > Please suggest if I need to fine tune it more. > > > > > > > > > Below is given our code for MINA configuration. > > > import java.io.IOException; > > > import java.net.InetSocketAddress; import org.apache.log4j.Logger; > > > import org.apache.mina.core.service.AbstractIoAcceptor; > > > import org.apache.mina.core.session.IdleStatus; > > > import org.apache.mina.filter.logging.LoggingFilter; > > > import org.apache.mina.transport.socket.nio.NioDatagramAcceptor; > > > > > > import com.techm.tnt.protocol.gateway.util.ServerConfigLoader; > > > > > > /** > > > * The abstract class for tcp/ip and udp listeners. > > > */ > > > public abstract class AbstractListener { > > > > > > /** The logger instance.*/ > > > private static Logger LOGGER = > > > Logger.getLogger(AbstractListener.class); > > > > > > /** The buffer size constant. */ > > > protected static final int BUFFER_SIZE = 2048; > > > > > > /** > > > * The method to initialize udp and tcp/ip listeners. > > > * > > > * @param portKey > > > * the port of the udp or tcp/ip listener. > > > * @throws IOException > > > * the io exception to be thrown in case of > > error. > > > */ > > > public void init(String portKey) throws IOException { > > > LOGGER.info("Entering init()"); > > > AbstractIoAcceptor acceptor = > > initInternal(); > > > acceptor.setHandler(new > > > RequestHandler()); > > > > > acceptor.getFilterChain().addLast("logger", new LoggingFilter()); > > > > > acceptor.getSessionConfig().setReadBufferSize(BUFFER_SIZE); > > > > > acceptor.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE, 10); > > > if (acceptor instanceof > > NioDatagramAcceptor) { > > > > > > ((NioDatagramAcceptor) > > acceptor).getSessionConfig().setReuseAddress(true); > > > } > > > acceptor.bind(new > > InetSocketAddress(Integer.parseInt(ServerConfigLoader.getInstance(). > > ge > > tProperty(portKey)))); > > > LOGGER.info("Exiting init()"); > > > } > > > > > > /** > > > * The abstract method to be over ridden by > > > appropriate > > listener (udp / > > > * tcp/ip) for invoking appropriate listener. > > > * > > > * @return the appropriate (udp/ tcp/ip) io acceptor. > > > */ > > > public abstract AbstractIoAcceptor initInternal(); > > > } > > > > > > Appreciate all tour help. > > > > > > Thanks & Regards, > > > Krishan Babbar > > > > > > > > > > > > ================================================================== > > > == == ====================================================== > > > Disclaimer: This message and the information contained herein is > > proprietary and confidential and subject to the Tech Mahindra policy > > statement, you may review the policy at > > http://www.techmahindra.com/Disclaimer.html externally > > http://tim.techmahindra.com/tim/disclaimer.html internally within > > TechMahindra. > > > ================================================================== > > > == == ===================================================== > > > > > > ================================================================== > > > == == ====================================================== > > > > > > Disclaimer: This message and the information contained herein is > > proprietary and confidential and subject to the Tech Mahindra policy > > statement, you may review the policy at > > http://www.techmahindra.com/Disclaimer.html < > > http://www.techmahindra.com/Disclaimer.html> externally > > http://tim.techmahindra.com/tim/disclaimer.html < > > http://tim.techmahindra.com/tim/disclaimer.html> internally within > > TechMahindra. > > > > > > ================================================================== > > > == == ====================================================== > > > -- > > CONFIDENTIALITY NOTICE: The contents of this email message and any > attachments are intended solely for the addressee(s) and may contain > confidential and/or privileged information and may be legally > protected from disclosure. > > ====================================================================== > ====================================================== > > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html < > http://www.techmahindra.com/Disclaimer.html> externally > http://tim.techmahindra.com/tim/disclaimer.html < > http://tim.techmahindra.com/tim/disclaimer.html> internally within > TechMahindra. > > > ====================================================================== > ====================================================== > -- CONFIDENTIALITY NOTICE: The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. ============================================================================================================================ Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html <http://www.techmahindra.com/Disclaimer.html> externally http://tim.techmahindra.com/tim/disclaimer.html <http://tim.techmahindra.com/tim/disclaimer.html> internally within TechMahindra. ============================================================================================================================
