Perform a heap dump to find out what’s going on Sendt fra min iPhone
> 5. des. 2018 kl. 07:36 skrev Krishan Babbar <[email protected]>: > > Any idea why file count is increasing day by day? Can we close any > socket/file handle manually? > > Date > > Time > > Open files count > > 2-Dec > > 16:08 PM > > 7047 > > 3-Dec > > 8:29 AM > > 7209 > > 3-Dec > > 14:45 PM > > 7289 > > 3-Dec > > 16:00 PM > > 7308 > > 3-Dec > > 16:11 PM > > 7310 > > 3-Dec > > 18:01 PM > > 7330 > > 3-Dec > > 20:00 PM > > 7358 > > 3-Dec > > 22:00 PM > > 7394 > > 4-Dec > > 8:25 AM > > 7501 > > 4-Dec > > 11:09 AM > > 7543 > > 4-Dec > > 12:00 PM > > 7551 > > 4-Dec > > 14:00 PM > > 7594 > > 4-Dec > > 16:00 PM > > 7614 > > 4-Dec > > 18:00 PM > > 7635 > > 4-Dec > > 20:00 PM > > 7668 > > 4-Dec > > 22:00 PM > > 7689 > > > > Thanks & Regards, > Krishan Babbar > > ============================================================================================================================ > Disclaimer: This message and the information contained herein is proprietary > and confidential and subject to the Tech Mahindra policy statement, you may > review the policy at http://www.techmahindra.com/Disclaimer.html externally > http://tim.techmahindra.com/tim/disclaimer.html internally within > TechMahindra. > =========================================================================================================================== > > From: Krishan Babbar > Sent: Monday, December 3, 2018 11:27 AM > To: [email protected] > Cc: Atul Kandhari <[email protected]> > Subject: RE: Getting "Too many open files" warnings11 > > > Thanks Jonathan, > > > > Ok, but I did not see any message like "acceptor was sleeping". > > > > I noticed that my open files count for my Java application is keep increasing > day by day as shown below. Why is it not decreasing again? > > What configuration am I missing? What can be the reason? > > > Date > > Time > > Open file count > > 30-Nov > > 20:00 PM > > 2273 > > 30-Nov > > 22:00 PM > > 2544 > > 1-Dec > > 12:00 AM > > 2790 > > 1-Dec > > 2:00 AM > > 2987 > > 1-Dec > > 4:00 AM > > 3283 > > 1-Dec > > 12:00 PM > > 4828 > > 1-Dec > > 14:00 PM > > 5200 > > 1-Dec > > 16:00 PM > > 5635 > > 2-Dec > > 10:17 AM > > 6992 > > 2-Dec > > 12:00 PM > > 7008 > > 2-Dec > > 14:02 PM > > 7024 > > 2-Dec > > 16:08 PM > > 7047 > > > > > > > In my RequestHandler, I have following settings, means setting session > timeout to 1800 seconds (30 Mins) > @Override > public void messageReceived(IoSession session, Object message) throws > Exception { > if (session != null) { > this.session = session; > this.session.getConfig().setBothIdleTime(1800); > LOGGER.info("Session Idle time in sec: " + > this.session.getConfig().getBothIdleTime()); > } > > this.message = message; > > } > > > > Yes, I can simulate multiple devices with lowering nlimit and test. What > would be the next? > > > > Thanks & Regards, > > Krishan Babbar > > > > ============================================================================================================================ > > Disclaimer: This message and the information contained herein is proprietary > and confidential and subject to the Tech Mahindra policy statement, you may > review the policy at http://www.techmahindra.com/Disclaimer.html externally > http://tim.techmahindra.com/tim/disclaimer.html internally within > TechMahindra. > > =========================================================================================================================== > > > > -----Original Message----- > From: Jonathan Valliere > <[email protected]<mailto:[email protected]>> > Sent: Friday, November 30, 2018 7:20 PM > To: [email protected]<mailto:[email protected]> > Cc: Atul Kandhari > <[email protected]<mailto:[email protected]>> > Subject: Re: Getting "Too many open files" warnings11 > > > > Mina is patched to prevent that exception from occurring in a cycle. That > > was a long time ago, your version is patched. You probably noticed a > > message saying the acceptor was sleeping. > > > > Do you have any kind of testing utility to create fake users for your > application? I would recommend lowering the nlimit and trying to reproduce > > entirely artificially. There shouldn’t be any correlation to > > > > On Fri, Nov 30, 2018 at 8:39 AM Krishan Babbar > <[email protected]<mailto:[email protected]>> > > wrote: > > > >> Hi Jonathan, > >> > >> Version - 2.0.16 > >> Java Application process was using around 1% of CPU and 3.4% of Memory. > >> > >> After mitigate, were you able to provide the solution? > >> I mean what would be the solution if you know? > >> > >> Thanks & Regards, > >> Krishan Babbar > >> > >> -----Original Message----- > >> From: Jonathan Valliere >> <[email protected]<mailto:[email protected]>> > >> Sent: Friday, November 30, 2018 6:56 PM > >> To: [email protected]<mailto:[email protected]> > >> Cc: Atul Kandhari >> <[email protected]<mailto:[email protected]>> > >> Subject: Re: Getting "Too many open files" warnings11 > >> > >> What version of Mina are you using? We added code to mitigate this > >> specific exemption a while back. > >> > >> Do you know how much cpu Java was consuming at that point in time? > >> > >> On Fri, Nov 30, 2018 at 8:24 AM Krishan Babbar < > >> [email protected]<mailto:[email protected]>> > >> wrote: > >> > >>> Thanks Stefan, > >>> > >>> It would be great if you could answers other questions too. > >>> -What do you think, why application stops receiving packets from devices? > >>> Is it only due to exception "Too many open files" or it can be due > >>> to any other reasons as well? Please guide. > >>> -Is my MINA configuration fine shown in below Java code? Please > >>> suggest if I need to fine tune it more. > >>> > >>> Thanks & Regards, > >>> Krishan Babbar > >>> > >>> > >>> > >>> ==================================================================== > >>> == ====================================================== > >>> Disclaimer: This message and the information contained herein is > >>> proprietary and confidential and subject to the Tech Mahindra policy > >>> statement, you may review the policy at > >>> http://www.techmahindra.com/Disclaimer.html externally > >>> http://tim.techmahindra.com/tim/disclaimer.html internally within > >>> TechMahindra. > >>> > >>> ==================================================================== > >>> == ===================================================== > >>> > >>> -----Original Message----- > >>> From: Stefan Magnus Landrø >>> <[email protected]<mailto:[email protected]>> > >>> Sent: Friday, November 30, 2018 5:59 PM > >>> To: [email protected]<mailto:[email protected]> > >>> Cc: Atul Kandhari >>> <[email protected]<mailto:[email protected]>> > >>> Subject: Re: Getting "Too many open files" warnings11 > >>> > >>> > >>> https://unix.stackexchange.com/questions/108603/do-changes-in-etc-se > >>> cu > >>> rity-limits-conf-require-a-reboot > >>> > >>> > >>> Sendt fra min iPhone > >>> > >>>> 30. nov. 2018 kl. 11:17 skrev Krishan Babbar > >>>> <[email protected] > >>>> : > >>>> > >>>> Dear All, > >>>> > >>>> I am working on an IoT project. I have created a Java application > >>>> using > >>> MINA library. All the devices are connecting to this Java > >>> application using TCP protocol. I am getting around 3000 packets per > >>> hour from about 50 devices as of now and it will increase soon. > >>>> > >>>> I am facing one issue on production environment. After running say > >>>> 1-2 > >>> days, my application stops receiving packets from all the devices > >>> but application process keep running. To make it working again, I > >>> need to restart my Java application. I noticed following warnings in logs. > >>>> Warnings > >>>> [NioSocketAcceptor-3] WARN > >>>> org.apache.mina.util.DefaultExceptionMonitor > >>> 30/11/2018 05:25:40 - Unexpected exception. > >>>> java.io.IOException: Too many open files > >>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > >>>> at sun.nio.ch > >>> .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) > >>>> at sun.nio.ch > >>> .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > >>>> at > >>> org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioSoc > >>> ke > >>> tAcceptor.java:194) > >>>> at > >>> org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioSoc > >>> ke > >>> tAcceptor.java:51) > >>>> at > >>> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.proc > >>> es > >>> sHandles(AbstractPollingIoAcceptor.java:544) > >>>> at > >>> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run( > >>> Ab > >>> stractPollingIoAcceptor.java:484) > >>>> at > >>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnab > >>> le > >>> .java:64) > >>>> at > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor > >>> .j > >>> ava:1142) > >>>> at > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > >>> java:617) > >>>> at java.lang.Thread.run(Thread.java:745) > >>>> > >>>> Due to above warnings in logs, yesterday, after doing some Google, > >>>> I > >>> changed file limits in "/etc/security/limits.conf" file to 100000 as > >>> shown below, but I did not restart my Java application. > >>>> * hard nofile 100000 > >>>> * soft nofile 100000 > >>>> > >>>> Today. I got the same warnings again and I was not getting any > >>>> packet > >>> from any of the devices. So I restarted my Java application and > >>> devices started sending packets again. > >>>> Now, my questions are > >>>> > >>>> - After application stopped receiving packets, I restarted my > >>> Java application today. Will it consider new file limits i.e. 100000 > >>> and not give Exceptions again? Or do I need to reboot machine as well? > >>>> > >>>> - What do you think, why application stops receiving packets > >>> from devices? Is it only due to exception "Too many open files" or > >>> it can be due to any other reasons as well? Please guide. > >>>> > >>>> - Is my MINA configuration fine shown in below Java code? > >>> Please suggest if I need to fine tune it more. > >>>> > >>>> > >>>> Below is given our code for MINA configuration. > >>>> import java.io.IOException; > >>>> import java.net.InetSocketAddress; import org.apache.log4j.Logger; > >>>> import org.apache.mina.core.service.AbstractIoAcceptor; > >>>> import org.apache.mina.core.session.IdleStatus; > >>>> import org.apache.mina.filter.logging.LoggingFilter; > >>>> import org.apache.mina.transport.socket.nio.NioDatagramAcceptor; > >>>> > >>>> import com.techm.tnt.protocol.gateway.util.ServerConfigLoader; > >>>> > >>>> /** > >>>> * The abstract class for tcp/ip and udp listeners. > >>>> */ > >>>> public abstract class AbstractListener { > >>>> > >>>> /** The logger instance.*/ > >>>> private static Logger LOGGER = > >>>> Logger.getLogger(AbstractListener.class); > >>>> > >>>> /** The buffer size constant. */ > >>>> protected static final int BUFFER_SIZE = 2048; > >>>> > >>>> /** > >>>> * The method to initialize udp and tcp/ip listeners. > >>>> * > >>>> * @param portKey > >>>> * the port of the udp or tcp/ip listener. > >>>> * @throws IOException > >>>> * the io exception to be thrown in case of > >>> error. > >>>> */ > >>>> public void init(String portKey) throws IOException { > >>>> LOGGER.info("Entering init()"); > >>>> AbstractIoAcceptor acceptor = > >>> initInternal(); > >>>> acceptor.setHandler(new > >>>> RequestHandler()); > >>>> > >>> acceptor.getFilterChain().addLast("logger", new LoggingFilter()); > >>>> > >>> acceptor.getSessionConfig().setReadBufferSize(BUFFER_SIZE); > >>>> > >>> acceptor.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE, 10); > >>>> if (acceptor instanceof > >>> NioDatagramAcceptor) { > >>>> > >>>> ((NioDatagramAcceptor) > >>> acceptor).getSessionConfig().setReuseAddress(true); > >>>> } > >>>> acceptor.bind(new > >>> InetSocketAddress(Integer.parseInt(ServerConfigLoader.getInstance(). > >>> ge > >>> tProperty(portKey)))); > >>>> LOGGER.info("Exiting init()"); > >>>> } > >>>> > >>>> /** > >>>> * The abstract method to be over ridden by > >>>> appropriate > >>> listener (udp / > >>>> * tcp/ip) for invoking appropriate listener. > >>>> * > >>>> * @return the appropriate (udp/ tcp/ip) io acceptor. > >>>> */ > >>>> public abstract AbstractIoAcceptor initInternal(); > >>>> } > >>>> > >>>> Appreciate all tour help. > >>>> > >>>> Thanks & Regards, > >>>> Krishan Babbar > >>>> > >>>> > >>>> > >>>> ================================================================== > >>>> == == ====================================================== > >>>> Disclaimer: This message and the information contained herein is > >>> proprietary and confidential and subject to the Tech Mahindra policy > >>> statement, you may review the policy at > >>> http://www.techmahindra.com/Disclaimer.html externally > >>> http://tim.techmahindra.com/tim/disclaimer.html internally within > >>> TechMahindra. > >>>> ================================================================== > >>>> == == ===================================================== > >>>> > >>>> ================================================================== > >>>> == == ====================================================== > >>>> > >>>> Disclaimer: This message and the information contained herein is > >>> proprietary and confidential and subject to the Tech Mahindra policy > >>> statement, you may review the policy at > >>> http://www.techmahindra.com/Disclaimer.html < > >>> http://www.techmahindra.com/Disclaimer.html> externally > >>> http://tim.techmahindra.com/tim/disclaimer.html < > >>> http://tim.techmahindra.com/tim/disclaimer.html> internally within > >>> TechMahindra. > >>>> > >>>> ================================================================== > >>>> == == ====================================================== > >>> > >> -- > >> > >> CONFIDENTIALITY NOTICE: The contents of this email message and any > >> attachments are intended solely for the addressee(s) and may contain > >> confidential and/or privileged information and may be legally > >> protected from disclosure. > >> > >> ====================================================================== > >> ====================================================== > >> > >> Disclaimer: This message and the information contained herein is > >> proprietary and confidential and subject to the Tech Mahindra policy > >> statement, you may review the policy at > >> http://www.techmahindra.com/Disclaimer.html < > >> http://www.techmahindra.com/Disclaimer.html> externally > >> http://tim.techmahindra.com/tim/disclaimer.html < > >> http://tim.techmahindra.com/tim/disclaimer.html> internally within > >> TechMahindra. > >> > >> > >> ====================================================================== > >> ====================================================== > >> > > -- > > > > CONFIDENTIALITY NOTICE: The contents of this email message and any > attachments are intended solely for the addressee(s) and may contain > confidential and/or privileged information and may be legally protected from > disclosure. > ============================================================================================================================ > > Disclaimer: This message and the information contained herein is proprietary > and confidential and subject to the Tech Mahindra policy statement, you may > review the policy at http://www.techmahindra.com/Disclaimer.html > <http://www.techmahindra.com/Disclaimer.html> externally > http://tim.techmahindra.com/tim/disclaimer.html > <http://tim.techmahindra.com/tim/disclaimer.html> internally within > TechMahindra. > > ============================================================================================================================
