RE: java.net.SocketException: Too many open files
> From: john.c.cartwri...@noaa.gov [mailto:john.c.cartwri...@noaa.gov] > Can someone please help to to understand what might cause such an > exception? File descriptor exhaustion - the process has run out of fds. Any i/o could use a file descriptor, whether that's socket to httpd, socket to database or access to a file. Naively, I'd expect lsof to show them - what makes you think it isn't? If you're lucky, you merely need to find the piece of code that's leaking resources and fix it - which I accept isn't always the easiest of jobs. If you're *un*lucky, it's load related and you've just plain run out. I'll leave the UNIX specialists to suggest ways of increasing the number of fds per process, but there have been some recent threads on here. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: java.net.SocketException: Too many open files
Here is the code
void acceptConnections() {
if( log.isDebugEnabled() )
log.debug("Accepting ajp connections on " + port);
while( running ) {
try{
MsgContext ep=createMsgContext(packetSize);
ep.setSource(this);
ep.setWorkerEnv( wEnv );
this.accept(ep);
if( !running ) break;
// Since this is a long-running connection, we don't care about the
small GC
//next line gac's
SocketConnection ajpConn=
new SocketConnection(this, ep);
tp.runIt( ajpConn );
}catch(Exception ex) {
if (running) log.warn("Exception executing accept" ,ex);
}
}
}
where org.apache.jk.core.JkHandler.java contains the createMsgContext
public MsgContext createMsgContext() {
return new MsgContext(8*1024);
}
which calls org.apache.jk.core.MsgContext
public MsgContext(int bsize) {
try {
c2b = new C2BConverter("iso-8859-1");
} catch(IOException iex) {
log.warn("Can't happen", iex);
}
jkIS = new JkInputStream(this, bsize);
}
which calls org.apache.jk.common.JkInputStream
public JkInputStream(MsgContext context, int bsize) {
mc = context;
bodyMsg = new MsgAjp(bsize);
outputMsg = new MsgAjp(bsize);
}
//which calls org.apache.jk.common.MsgAjp
private byte buf[];
// The current read or write position in the buffer
private int pos;
/**
* This actually means different things depending on whether the
* packet is read or write. For read, it's the length of the
* payload (excluding the header). For write, it's the length of
* the packet as a whole (counting the header). Oh, well.
*/
private int len;
/*** The maximum packet size*/
private int bufsize;
/*** Constructor that takes a buffer size*/
public MsgAjp(int bsize) {
if(bsize < 8*1024) {
bsize = 8*1024;
}
bufsize = bsize;
buf = new byte[bsize];
}
3 possiblities
1)misconfig with AJP connector
2)so you're running out of heap
tweaking JAVA_OPT minheap and maxheap parameters would help
a 2004 post http://forums.sun.com/thread.jspa?messageID=10055131 says it best
The default stack size is 256K on UNIX and on 32-bit Windows operating systems.
To set Stack
size
java -XX:NewSize=128m -XX:MaxNewSize=128m -XX:SurvivorRatio=8 -Xms512m -Xmx512m
1. Setting the New generation heap size
-XX:NewSize
Use
this option to set the New generation Java heap size. Set this value to
a multiple of 1024 that is greater than 1MB. As a general rule, set
-XX:NewSize to be one-fourth the size of the maximum heap size.
Increase the value of this option for larger numbers of short-lived
objects.(problem is you would have to know which objects are short-lived)
Be sure to increase the New generation as you increase the number of
processors. Memory allocation can be parallel, but garbage collection
is not parallel.
2. Setting the maximum New generation heap size
-XX:MaxNewSize
Use this option to set the maximum New generation Java heap size. Set
this value to a multiple of 1024 that is greater than 1MB.
(this is the ceiling for New Generation heap)
3. Setting New heap size ratios
-XX:SurvivorRatio
The New generation area is divided into three sub-areas: Eden, and two
survivor spaces that
are equal in size.
Use the -XX:SurvivorRatio=X option to configure the ratio of the
Eden/survivor space size. Try setting this value to 8, and then monitor
your garbage collection.
4. Setting minimum heap size
-Xms
Use this option to set the minimum size of the memory allocation pool.
Set this value to a multiple of 1024 that is greater than 1MB. As a
general rule, set minimum heap size (-Xms) equal to the maximum heap
size (-Xmx) to minimize garbage collections.
5. Setting maximum heap size
-Xmx
Use this option to set the maximum Java heap size. Set this value to a
multiple of 1024 that
is greater than 1MB.
i would double all specified params (leave Survivor alone) until maxheap -Xmx
reaches AvailablePhysicalRam/2
2)you've maxed out HW RAM
at which point I'd run down to local tech supply store and buy more RAM sticks
(good to wait for offhours
,down the server and take one of the ram sticks with you just to make sure you
have a good fit..generally safe to double..if you have 1 get another..if you
have 2 get 2 more )
(unless of course this is a unisys box then i would ask Unisys)
Martin Gainty
__
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger
sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung
oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem
Re: java.net.SocketException: Too many open files
Thanks for your suggestions Martin, I'll look into modifying the memory
parameters. Strange thing is that this has been running for weeks w/o
any changes in the configuration or contexts.
--john
Martin Gainty wrote:
Here is the code
void acceptConnections() {
if( log.isDebugEnabled() )
log.debug("Accepting ajp connections on " + port);
while( running ) {
try{
MsgContext ep=createMsgContext(packetSize);
ep.setSource(this);
ep.setWorkerEnv( wEnv );
this.accept(ep);
if( !running ) break;
// Since this is a long-running connection, we don't care about the small GC
//next line gac's
SocketConnection ajpConn=
new SocketConnection(this, ep);
tp.runIt( ajpConn );
}catch(Exception ex) {
if (running) log.warn("Exception executing accept" ,ex);
}
}
}
where org.apache.jk.core.JkHandler.java contains the createMsgContext
public MsgContext createMsgContext() {
return new MsgContext(8*1024);
}
which calls org.apache.jk.core.MsgContext
public MsgContext(int bsize) {
try {
c2b = new C2BConverter("iso-8859-1");
} catch(IOException iex) {
log.warn("Can't happen", iex);
}
jkIS = new JkInputStream(this, bsize);
}
which calls org.apache.jk.common.JkInputStream
public JkInputStream(MsgContext context, int bsize) {
mc = context;
bodyMsg = new MsgAjp(bsize);
outputMsg = new MsgAjp(bsize);
}
//which calls org.apache.jk.common.MsgAjp
private byte buf[];
// The current read or write position in the buffer
private int pos;
/**
* This actually means different things depending on whether the
* packet is read or write. For read, it's the length of the
* payload (excluding the header). For write, it's the length of
* the packet as a whole (counting the header). Oh, well.
*/
private int len;
/*** The maximum packet size*/
private int bufsize;
/*** Constructor that takes a buffer size*/
public MsgAjp(int bsize) {
if(bsize < 8*1024) {
bsize = 8*1024;
}
bufsize = bsize;
buf = new byte[bsize];
}
3 possiblities
1)misconfig with AJP connector
2)so you're running out of heap
tweaking JAVA_OPT minheap and maxheap parameters would help
a 2004 post http://forums.sun.com/thread.jspa?messageID=10055131 says it best
The default stack size is 256K on UNIX and on 32-bit Windows operating systems.
To set Stack
size
java -XX:NewSize=128m -XX:MaxNewSize=128m -XX:SurvivorRatio=8 -Xms512m -Xmx512m
1. Setting the New generation heap size
-XX:NewSize
Use
this option to set the New generation Java heap size. Set this value to
a multiple of 1024 that is greater than 1MB. As a general rule, set
-XX:NewSize to be one-fourth the size of the maximum heap size.
Increase the value of this option for larger numbers of short-lived
objects.(problem is you would have to know which objects are short-lived)
Be sure to increase the New generation as you increase the number of
processors. Memory allocation can be parallel, but garbage collection
is not parallel.
2. Setting the maximum New generation heap size
-XX:MaxNewSize
Use this option to set the maximum New generation Java heap size. Set
this value to a multiple of 1024 that is greater than 1MB.
(this is the ceiling for New Generation heap)
3. Setting New heap size ratios
-XX:SurvivorRatio
The New generation area is divided into three sub-areas: Eden, and two
survivor spaces that
are equal in size.
Use the -XX:SurvivorRatio=X option to configure the ratio of the
Eden/survivor space size. Try setting this value to 8, and then monitor
your garbage collection.
4. Setting minimum heap size
-Xms
Use this option to set the minimum size of the memory allocation pool.
Set this value to a multiple of 1024 that is greater than 1MB. As a
general rule, set minimum heap size (-Xms) equal to the maximum heap
size (-Xmx) to minimize garbage collections.
5. Setting maximum heap size
-Xmx
Use this option to set the maximum Java heap size. Set this value to a
multiple of 1024 that
is greater than 1MB.
i would double all specified params (leave Survivor alone) until maxheap -Xmx
reaches AvailablePhysicalRam/2
2)you've maxed out HW RAM
at which point I'd run down to local tech supply store and buy more RAM sticks
(good to wait for offhours
,down the server and take one of the ram sticks with you just to make sure you
have a good fit..generally safe to double..if you have 1 get another..if you
have 2 get 2 more )
(unless of course this is a unisys box then i would ask Unisys)
Martin Gainty
__
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Diese Nac
RE: java.net.SocketException: Too many open files
> From: John Cartwright [mailto:john.c.cartwri...@noaa.gov] > Subject: Re: java.net.SocketException: Too many open files > > I'll look into modifying the memory parameters. That would be a complete waste of time. The heap size has nothing to do with the problem you're seeing, and the discussion cited by Martin is full of erroneous information. Please ignore it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: java.net.SocketException: Too many open files
Thanks for your reply Peter. Initially I was assuming that lsof was not showing me files on disk that were being opened and read by servlets. However, I've been unable to reproduce that in a more controlled setting. Since this system has been running for weeks w/o any modification, something must be leaking. I'll keep monitoring and see if I can gather more information. --john Peter Crowther wrote: From: john.c.cartwri...@noaa.gov [mailto:john.c.cartwri...@noaa.gov] Can someone please help to to understand what might cause such an exception? File descriptor exhaustion - the process has run out of fds. Any i/o could use a file descriptor, whether that's socket to httpd, socket to database or access to a file. Naively, I'd expect lsof to show them - what makes you think it isn't? If you're lucky, you merely need to find the piece of code that's leaking resources and fix it - which I accept isn't always the easiest of jobs. If you're *un*lucky, it's load related and you've just plain run out. I'll leave the UNIX specialists to suggest ways of increasing the number of fds per process, but there have been some recent threads on here. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: java.net.SocketException: Too many open files
glad to hear that helped
you may want to consider implementing SSL on your connectors to protect your TC
resources
http://www.mbaworld.com/docs/ssl-howto.html
Please let us know if we can provide any assistance to your efforts
Martin Gainty
__
Jogi és Bizalmassági kinyilatkoztatás/Verzicht und
Vertraulichkeitanmerkung/Note de déni et de confidentialité
Ez az
üzenet bizalmas. Ha nem ön az akinek szánva volt, akkor kérjük, hogy
jelentse azt nekünk vissza. Semmiféle továbbítása vagy másolatának
készítése nem megengedett. Ez az üzenet csak ismeret cserét szolgál és
semmiféle jogi alkalmazhatósága sincs. Mivel az electronikus üzenetek
könnyen megváltoztathatóak, ezért minket semmi felelöség nem terhelhet
ezen üzenet tartalma miatt.
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger
sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung
oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem
Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung.
Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung
fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le
destinataire prévu, nous te demandons avec bonté que pour satisfaire informez
l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est
interdite. Ce message sert à l'information seulement et n'aura pas n'importe
quel effet légalement obligatoire. Étant donné que les email peuvent facilement
être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité
pour le contenu fourni.
> Date: Wed, 3 Jun 2009 14:15:43 -0600
> From: john.c.cartwri...@noaa.gov
> Subject: Re: java.net.SocketException: Too many open files
> To: users@tomcat.apache.org
>
> Thanks for your suggestions Martin, I'll look into modifying the memory
> parameters. Strange thing is that this has been running for weeks w/o
> any changes in the configuration or contexts.
>
> --john
>
>
> Martin Gainty wrote:
> > Here is the code
> > void acceptConnections() {
> > if( log.isDebugEnabled() )
> > log.debug("Accepting ajp connections on " + port);
> > while( running ) {
> > try{
> > MsgContext ep=createMsgContext(packetSize);
> > ep.setSource(this);
> > ep.setWorkerEnv( wEnv );
> > this.accept(ep);
> >
> > if( !running ) break;
> >
> > // Since this is a long-running connection, we don't care about
> > the small GC
> > //next line gac's
> > SocketConnection ajpConn=
> > new SocketConnection(this, ep);
> > tp.runIt( ajpConn );
> > }catch(Exception ex) {
> > if (running) log.warn("Exception executing accept" ,ex);
> > }
> > }
> > }
> >
> > where org.apache.jk.core.JkHandler.java contains the createMsgContext
> > public MsgContext createMsgContext() {
> > return new MsgContext(8*1024);
> > }
> >
> > which calls org.apache.jk.core.MsgContext
> > public MsgContext(int bsize) {
> > try {
> > c2b = new C2BConverter("iso-8859-1");
> > } catch(IOException iex) {
> > log.warn("Can't happen", iex);
> > }
> > jkIS = new JkInputStream(this, bsize);
> > }
> >
> > which calls org.apache.jk.common.JkInputStream
> > public JkInputStream(MsgContext context, int bsize) {
> > mc = context;
> > bodyMsg = new MsgAjp(bsize);
> > outputMsg = new MsgAjp(bsize);
> > }
> >
> > //which calls org.apache.jk.common.MsgAjp
> > private byte buf[];
> > // The current read or write position in the buffer
> > private int pos;
> > /**
> > * This actually means different things depending on whether the
> > * packet is read or write. For read, it's the length of the
> > * payload (excluding the header). For write, it's the length of
> > * the packet as a whole (counting the header). Oh, well.
> > */
> > private int len;
> > /*** The maximum packet size*/
> > private int bufsize;
> >
> > /*** Constructor that takes a buffer size*/
> > public MsgAjp(int bsize) {
> > if(bsize < 8*1024) {
> > bsize = 8*1024;
RE: java.net.SocketException: Too many open files
Greetings All, I had the exact same issue at one time a while ago, my resolution was the setting of the max number of files that was aloowed to be opened by the OS which affected the tomcat application. After I increased the max-files setting, all was resolved. Kind regards / Met vriendelijke groet, Lawrence Lamprecht -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Wednesday, June 03, 2009 10:20 PM To: Tomcat Users List Subject: RE: java.net.SocketException: Too many open files > From: John Cartwright [mailto:john.c.cartwri...@noaa.gov] > Subject: Re: java.net.SocketException: Too many open files > > I'll look into modifying the memory parameters. That would be a complete waste of time. The heap size has nothing to do with the problem you're seeing, and the discussion cited by Martin is full of erroneous information. Please ignore it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
