RE: java.net.SocketException: Too many open files
> From: john.c.cartwri...@noaa.gov [mailto:john.c.cartwri...@noaa.gov] > Can someone please help to to understand what might cause such an > exception? File descriptor exhaustion - the process has run out of fds. Any i/o could use a file descriptor, whether that's socket to httpd, socket to database or access to a file. Naively, I'd expect lsof to show them - what makes you think it isn't? If you're lucky, you merely need to find the piece of code that's leaking resources and fix it - which I accept isn't always the easiest of jobs. If you're *un*lucky, it's load related and you've just plain run out. I'll leave the UNIX specialists to suggest ways of increasing the number of fds per process, but there have been some recent threads on here. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: java.net.SocketException: Too many open files
Here is the code void acceptConnections() { if( log.isDebugEnabled() ) log.debug("Accepting ajp connections on " + port); while( running ) { try{ MsgContext ep=createMsgContext(packetSize); ep.setSource(this); ep.setWorkerEnv( wEnv ); this.accept(ep); if( !running ) break; // Since this is a long-running connection, we don't care about the small GC //next line gac's SocketConnection ajpConn= new SocketConnection(this, ep); tp.runIt( ajpConn ); }catch(Exception ex) { if (running) log.warn("Exception executing accept" ,ex); } } } where org.apache.jk.core.JkHandler.java contains the createMsgContext public MsgContext createMsgContext() { return new MsgContext(8*1024); } which calls org.apache.jk.core.MsgContext public MsgContext(int bsize) { try { c2b = new C2BConverter("iso-8859-1"); } catch(IOException iex) { log.warn("Can't happen", iex); } jkIS = new JkInputStream(this, bsize); } which calls org.apache.jk.common.JkInputStream public JkInputStream(MsgContext context, int bsize) { mc = context; bodyMsg = new MsgAjp(bsize); outputMsg = new MsgAjp(bsize); } //which calls org.apache.jk.common.MsgAjp private byte buf[]; // The current read or write position in the buffer private int pos; /** * This actually means different things depending on whether the * packet is read or write. For read, it's the length of the * payload (excluding the header). For write, it's the length of * the packet as a whole (counting the header). Oh, well. */ private int len; /*** The maximum packet size*/ private int bufsize; /*** Constructor that takes a buffer size*/ public MsgAjp(int bsize) { if(bsize < 8*1024) { bsize = 8*1024; } bufsize = bsize; buf = new byte[bsize]; } 3 possiblities 1)misconfig with AJP connector 2)so you're running out of heap tweaking JAVA_OPT minheap and maxheap parameters would help a 2004 post http://forums.sun.com/thread.jspa?messageID=10055131 says it best The default stack size is 256K on UNIX and on 32-bit Windows operating systems. To set Stack size java -XX:NewSize=128m -XX:MaxNewSize=128m -XX:SurvivorRatio=8 -Xms512m -Xmx512m 1. Setting the New generation heap size -XX:NewSize Use this option to set the New generation Java heap size. Set this value to a multiple of 1024 that is greater than 1MB. As a general rule, set -XX:NewSize to be one-fourth the size of the maximum heap size. Increase the value of this option for larger numbers of short-lived objects.(problem is you would have to know which objects are short-lived) Be sure to increase the New generation as you increase the number of processors. Memory allocation can be parallel, but garbage collection is not parallel. 2. Setting the maximum New generation heap size -XX:MaxNewSize Use this option to set the maximum New generation Java heap size. Set this value to a multiple of 1024 that is greater than 1MB. (this is the ceiling for New Generation heap) 3. Setting New heap size ratios -XX:SurvivorRatio The New generation area is divided into three sub-areas: Eden, and two survivor spaces that are equal in size. Use the -XX:SurvivorRatio=X option to configure the ratio of the Eden/survivor space size. Try setting this value to 8, and then monitor your garbage collection. 4. Setting minimum heap size -Xms Use this option to set the minimum size of the memory allocation pool. Set this value to a multiple of 1024 that is greater than 1MB. As a general rule, set minimum heap size (-Xms) equal to the maximum heap size (-Xmx) to minimize garbage collections. 5. Setting maximum heap size -Xmx Use this option to set the maximum Java heap size. Set this value to a multiple of 1024 that is greater than 1MB. i would double all specified params (leave Survivor alone) until maxheap -Xmx reaches AvailablePhysicalRam/2 2)you've maxed out HW RAM at which point I'd run down to local tech supply store and buy more RAM sticks (good to wait for offhours ,down the server and take one of the ram sticks with you just to make sure you have a good fit..generally safe to double..if you have 1 get another..if you have 2 get 2 more ) (unless of course this is a unisys box then i would ask Unisys) Martin Gainty __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem
Re: java.net.SocketException: Too many open files
Thanks for your suggestions Martin, I'll look into modifying the memory parameters. Strange thing is that this has been running for weeks w/o any changes in the configuration or contexts. --john Martin Gainty wrote: Here is the code void acceptConnections() { if( log.isDebugEnabled() ) log.debug("Accepting ajp connections on " + port); while( running ) { try{ MsgContext ep=createMsgContext(packetSize); ep.setSource(this); ep.setWorkerEnv( wEnv ); this.accept(ep); if( !running ) break; // Since this is a long-running connection, we don't care about the small GC //next line gac's SocketConnection ajpConn= new SocketConnection(this, ep); tp.runIt( ajpConn ); }catch(Exception ex) { if (running) log.warn("Exception executing accept" ,ex); } } } where org.apache.jk.core.JkHandler.java contains the createMsgContext public MsgContext createMsgContext() { return new MsgContext(8*1024); } which calls org.apache.jk.core.MsgContext public MsgContext(int bsize) { try { c2b = new C2BConverter("iso-8859-1"); } catch(IOException iex) { log.warn("Can't happen", iex); } jkIS = new JkInputStream(this, bsize); } which calls org.apache.jk.common.JkInputStream public JkInputStream(MsgContext context, int bsize) { mc = context; bodyMsg = new MsgAjp(bsize); outputMsg = new MsgAjp(bsize); } //which calls org.apache.jk.common.MsgAjp private byte buf[]; // The current read or write position in the buffer private int pos; /** * This actually means different things depending on whether the * packet is read or write. For read, it's the length of the * payload (excluding the header). For write, it's the length of * the packet as a whole (counting the header). Oh, well. */ private int len; /*** The maximum packet size*/ private int bufsize; /*** Constructor that takes a buffer size*/ public MsgAjp(int bsize) { if(bsize < 8*1024) { bsize = 8*1024; } bufsize = bsize; buf = new byte[bsize]; } 3 possiblities 1)misconfig with AJP connector 2)so you're running out of heap tweaking JAVA_OPT minheap and maxheap parameters would help a 2004 post http://forums.sun.com/thread.jspa?messageID=10055131 says it best The default stack size is 256K on UNIX and on 32-bit Windows operating systems. To set Stack size java -XX:NewSize=128m -XX:MaxNewSize=128m -XX:SurvivorRatio=8 -Xms512m -Xmx512m 1. Setting the New generation heap size -XX:NewSize Use this option to set the New generation Java heap size. Set this value to a multiple of 1024 that is greater than 1MB. As a general rule, set -XX:NewSize to be one-fourth the size of the maximum heap size. Increase the value of this option for larger numbers of short-lived objects.(problem is you would have to know which objects are short-lived) Be sure to increase the New generation as you increase the number of processors. Memory allocation can be parallel, but garbage collection is not parallel. 2. Setting the maximum New generation heap size -XX:MaxNewSize Use this option to set the maximum New generation Java heap size. Set this value to a multiple of 1024 that is greater than 1MB. (this is the ceiling for New Generation heap) 3. Setting New heap size ratios -XX:SurvivorRatio The New generation area is divided into three sub-areas: Eden, and two survivor spaces that are equal in size. Use the -XX:SurvivorRatio=X option to configure the ratio of the Eden/survivor space size. Try setting this value to 8, and then monitor your garbage collection. 4. Setting minimum heap size -Xms Use this option to set the minimum size of the memory allocation pool. Set this value to a multiple of 1024 that is greater than 1MB. As a general rule, set minimum heap size (-Xms) equal to the maximum heap size (-Xmx) to minimize garbage collections. 5. Setting maximum heap size -Xmx Use this option to set the maximum Java heap size. Set this value to a multiple of 1024 that is greater than 1MB. i would double all specified params (leave Survivor alone) until maxheap -Xmx reaches AvailablePhysicalRam/2 2)you've maxed out HW RAM at which point I'd run down to local tech supply store and buy more RAM sticks (good to wait for offhours ,down the server and take one of the ram sticks with you just to make sure you have a good fit..generally safe to double..if you have 1 get another..if you have 2 get 2 more ) (unless of course this is a unisys box then i would ask Unisys) Martin Gainty __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nac
RE: java.net.SocketException: Too many open files
> From: John Cartwright [mailto:john.c.cartwri...@noaa.gov] > Subject: Re: java.net.SocketException: Too many open files > > I'll look into modifying the memory parameters. That would be a complete waste of time. The heap size has nothing to do with the problem you're seeing, and the discussion cited by Martin is full of erroneous information. Please ignore it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.net.SocketException: Too many open files
Thanks for your reply Peter. Initially I was assuming that lsof was not showing me files on disk that were being opened and read by servlets. However, I've been unable to reproduce that in a more controlled setting. Since this system has been running for weeks w/o any modification, something must be leaking. I'll keep monitoring and see if I can gather more information. --john Peter Crowther wrote: From: john.c.cartwri...@noaa.gov [mailto:john.c.cartwri...@noaa.gov] Can someone please help to to understand what might cause such an exception? File descriptor exhaustion - the process has run out of fds. Any i/o could use a file descriptor, whether that's socket to httpd, socket to database or access to a file. Naively, I'd expect lsof to show them - what makes you think it isn't? If you're lucky, you merely need to find the piece of code that's leaking resources and fix it - which I accept isn't always the easiest of jobs. If you're *un*lucky, it's load related and you've just plain run out. I'll leave the UNIX specialists to suggest ways of increasing the number of fds per process, but there have been some recent threads on here. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: java.net.SocketException: Too many open files
glad to hear that helped you may want to consider implementing SSL on your connectors to protect your TC resources http://www.mbaworld.com/docs/ssl-howto.html Please let us know if we can provide any assistance to your efforts Martin Gainty __ Jogi és Bizalmassági kinyilatkoztatás/Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Ez az üzenet bizalmas. Ha nem ön az akinek szánva volt, akkor kérjük, hogy jelentse azt nekünk vissza. Semmiféle továbbítása vagy másolatának készítése nem megengedett. Ez az üzenet csak ismeret cserét szolgál és semmiféle jogi alkalmazhatósága sincs. Mivel az electronikus üzenetek könnyen megváltoztathatóak, ezért minket semmi felelöség nem terhelhet ezen üzenet tartalma miatt. Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. > Date: Wed, 3 Jun 2009 14:15:43 -0600 > From: john.c.cartwri...@noaa.gov > Subject: Re: java.net.SocketException: Too many open files > To: users@tomcat.apache.org > > Thanks for your suggestions Martin, I'll look into modifying the memory > parameters. Strange thing is that this has been running for weeks w/o > any changes in the configuration or contexts. > > --john > > > Martin Gainty wrote: > > Here is the code > > void acceptConnections() { > > if( log.isDebugEnabled() ) > > log.debug("Accepting ajp connections on " + port); > > while( running ) { > > try{ > > MsgContext ep=createMsgContext(packetSize); > > ep.setSource(this); > > ep.setWorkerEnv( wEnv ); > > this.accept(ep); > > > > if( !running ) break; > > > > // Since this is a long-running connection, we don't care about > > the small GC > > //next line gac's > > SocketConnection ajpConn= > > new SocketConnection(this, ep); > > tp.runIt( ajpConn ); > > }catch(Exception ex) { > > if (running) log.warn("Exception executing accept" ,ex); > > } > > } > > } > > > > where org.apache.jk.core.JkHandler.java contains the createMsgContext > > public MsgContext createMsgContext() { > > return new MsgContext(8*1024); > > } > > > > which calls org.apache.jk.core.MsgContext > > public MsgContext(int bsize) { > > try { > > c2b = new C2BConverter("iso-8859-1"); > > } catch(IOException iex) { > > log.warn("Can't happen", iex); > > } > > jkIS = new JkInputStream(this, bsize); > > } > > > > which calls org.apache.jk.common.JkInputStream > > public JkInputStream(MsgContext context, int bsize) { > > mc = context; > > bodyMsg = new MsgAjp(bsize); > > outputMsg = new MsgAjp(bsize); > > } > > > > //which calls org.apache.jk.common.MsgAjp > > private byte buf[]; > > // The current read or write position in the buffer > > private int pos; > > /** > > * This actually means different things depending on whether the > > * packet is read or write. For read, it's the length of the > > * payload (excluding the header). For write, it's the length of > > * the packet as a whole (counting the header). Oh, well. > > */ > > private int len; > > /*** The maximum packet size*/ > > private int bufsize; > > > > /*** Constructor that takes a buffer size*/ > > public MsgAjp(int bsize) { > > if(bsize < 8*1024) { > > bsize = 8*1024;
RE: java.net.SocketException: Too many open files
Greetings All, I had the exact same issue at one time a while ago, my resolution was the setting of the max number of files that was aloowed to be opened by the OS which affected the tomcat application. After I increased the max-files setting, all was resolved. Kind regards / Met vriendelijke groet, Lawrence Lamprecht -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Wednesday, June 03, 2009 10:20 PM To: Tomcat Users List Subject: RE: java.net.SocketException: Too many open files > From: John Cartwright [mailto:john.c.cartwri...@noaa.gov] > Subject: Re: java.net.SocketException: Too many open files > > I'll look into modifying the memory parameters. That would be a complete waste of time. The heap size has nothing to do with the problem you're seeing, and the discussion cited by Martin is full of erroneous information. Please ignore it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org