RE: [OT] Re: linux : Limit of file descriptor
Francois, If you've solved this problem, please disregard this message. I've marked this issue as off topic because it relates to Linux and not Tomcat. Well, counting FDs is linux related, but why is there so much FDs opened could be tomcat related (even if, as a bofh, I would say it's something to do with those thing, you know? That specific sort of users called developers). Anyway. lsof won't properly report the total number of opened file descriptors. I was doing the same thing for a client who ran into this issue with file- max set to 16384. I was using lsof and wc to count the lines, however it reported something like 80,000. Yes, I ran into this lately yesterday (GMT), lsof reported a number of FDs of more than 22, so as already said, it should report shared FDs as well. Please apologize. You sound pretty knowledgeable, so you may have seen this, however the best way to handle this is using the proc file system. An explaination is available at http://www.ussg.iu.edu/hypermail/linux/kernel/0208.2/0145.html Yep, I use it. Can you post the contents of the following files: /proc/sys/fs/file-max As already mentioned :209713 /proc/sys/fs/file-nr 23321398209713 So. 2332 FDs opened, 1398 used. I found a message talking about a hard coded limit of 1024 in the Jvm... Is it a per thread limit or ? /proc/sys/fs/inode-nr 164972 39841 It seems to be OK there, about 4 inodes free /proc/sys/fs/inode-state 165000 39823 0 0 0 0 0 Show about the same, just adding everything is OK, good luck, and see you later in a OS point of view. Once you've pasted this information, we'll see if we can figure this out. So. I think I ran into a Jvm limit (user is unlimited, shame on me). I will go back to the developers corner to ask. Thank you for your help. François. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT] Re: linux : Limit of file descriptor
Francois: We have not run into a FD limit within the JVM using Tomcat, (only aformentioned user limits that can be modified using ulimit). I too have read of OS/JVM limits surrounding some operations like select() for the NIO package, but these apparently are not limits on Tomcat or Coyote Connector implementations AFAIK. Randy Watler Finali Corporation Francois JEANMOUGIN wrote: Can you post the contents of the following files: /proc/sys/fs/file-max As already mentioned :209713 /proc/sys/fs/file-nr 23321398209713 So. 2332 FDs opened, 1398 used. I found a message talking about a hard coded limit of 1024 in the Jvm... Is it a per thread limit or ?
linux : Limit of file descriptor
Hi all, One of my developers reached the limit of file descriptors using tomcat. With no more than 400 clients at a time and 500 threads, we have more than 20 (two hundred thousands) of FD opened. As far as I can read (just try to read a lsof output with more than 20 lines), each thread seems to open all the jar and java files. Anyone already seen this? It only happens on this machine, so it is probably configuration or application related. Just need some tip about such problem. Thanks, François. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: linux : Limit of file descriptor
Those are shared fd's. (Otherwise you would have hit the limit much earlier as it is typically something betweenn 1024 and 4096). -Original Message- From: Francois JEANMOUGIN [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 22, 2003 3:05 PM To: Tomcat Users List Subject: linux : Limit of file descriptor One of my developers reached the limit of file descriptors using tomcat. With no more than 400 clients at a time and 500 threads, we have more than 20 (two hundred thousands) of FD opened. As far as I can read (just try to read a lsof output with more than 20 lines), each thread seems to open all the jar and java files. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: linux : Limit of file descriptor
-Message d'origine- De : Ralph Einfeldt [mailto:[EMAIL PROTECTED] Envoyé : mercredi 22 octobre 2003 15:09 À : Tomcat Users List Objet : RE: linux : Limit of file descriptor Those are shared fd's. Nope, those are real FDs (Otherwise you would have hit the limit much earlier as it is typically something betweenn 1024 and 4096). Nope, on this server, it is 209713. You want some? François. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: linux : Limit of file descriptor
Francois, For what it is worth, we run vanilla RedHat 8.0 servers and have to bump up the number handles available to Tomcat using ulimit, (otherwise it is limited to ~1024). We have a complex application that does a lot of proxied access work. Despite that, we have only run out of file descriptors once in several years during what we think was a DOS attack. Remember, each socket gets a file descriptor as well. It seems unlikely that the FDs are being consumed by Tomcat if your log files look normal. Does the number of file handles climb steadily over time or rapidly balloon beyond norms on startup? What build of Linux is on the box in question? Randy Watler Finali Corporation Francois JEANMOUGIN wrote: -Message d'origine- De : Ralph Einfeldt [mailto:[EMAIL PROTECTED] Envoyé : mercredi 22 octobre 2003 15:09 À : Tomcat Users List Objet : RE: linux : Limit of file descriptor Those are shared fd's. Nope, those are real FDs (Otherwise you would have hit the limit much earlier as it is typically something betweenn 1024 and 4096). Nope, on this server, it is 209713. You want some? François. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] smime.p7s Description: S/MIME Cryptographic Signature
RE: linux : Limit of file descriptor
For what it is worth, we run vanilla RedHat 8.0 servers and have to bump up the number handles available to Tomcat using ulimit, (otherwise it is limited to ~1024). We have a complex application that does a lot of proxied access work. Despite that, we have only run out of file descriptors once in several years during what we think was a DOS attack. Remember, each socket gets a file descriptor as well. Yes. So we have 500 clients and this uses a pool of SQL connections (about 120). Everything else is local open files. Mostly jar and java that doesn't seem to be shared. That's strange. It seems unlikely that the FDs are being consumed by Tomcat if your log files look normal. Does the number of file handles climb steadily over time or rapidly balloon beyond norms on startup? What build of Linux is on the box in question? It is a RedHat enterprise server 2.1 (due to silly constructor support restrictions). I just saw the problem today, so I don't saw it coming up. The service was restarted this morning and two hours later, it already reached the current score. This night, it will be dead... On another machine running about the same type of applications (and two tomcat instances), I only have 8000FDs opened (which could be correcte regarding the number of jars, connections and log files). François. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: linux : Limit of file descriptor
Francois: H. Are you running with a fronting web server, (i.e. Apache, IIS, etc), or Tomcat stand alone? Randy Watler Finali Corporation smime.p7s Description: S/MIME Cryptographic Signature
RE: linux : Limit of file descriptor
Francois: H. Are you running with a fronting web server, (i.e. Apache, IIS, etc), or Tomcat stand alone? Standalone. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: linux : Limit of file descriptor
Sorry, but I can't beleaf that each thread has his own fd. On my box I have a limit of 1024 files but lsof report over 6500 open fd's. (As the the jar files are memory mapped files on my box, lsof doesn't show the fd, so I can't proove that they are shared) -Original Message- From: Francois JEANMOUGIN [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 22, 2003 3:25 PM To: Tomcat Users List Subject: RE: linux : Limit of file descriptor Those are shared fd's. Nope, those are real FDs - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: linux : Limit of file descriptor
Francois, This may be a Red Herring, but I have to wonder if the usage of Posix threads in the RedHat enterprise servers is causing problems somehow? Are your other servers RedHat enterprise boxen or different in other ways? What JVM are you using? Randy Watler Finali Corporation smime.p7s Description: S/MIME Cryptographic Signature
[OT] Re: linux : Limit of file descriptor
Quoting Randy Watler [EMAIL PROTECTED]: Francois, This may be a Red Herring, but I have to wonder if the usage of Posix threads in the RedHat enterprise servers is causing problems somehow? Are your other servers RedHat enterprise boxen or different in other ways? What JVM are you using? Randy Watler Finali Corporation Francois, If you've solved this problem, please disregard this message. I've marked this issue as off topic because it relates to Linux and not Tomcat. lsof won't properly report the total number of opened file descriptors. I was doing the same thing for a client who ran into this issue with file-max set to 16384. I was using lsof and wc to count the lines, however it reported something like 80,000. You sound pretty knowledgeable, so you may have seen this, however the best way to handle this is using the proc file system. An explaination is available at http://www.ussg.iu.edu/hypermail/linux/kernel/0208.2/0145.html Can you post the contents of the following files: /proc/sys/fs/file-max /proc/sys/fs/file-nr /proc/sys/fs/inode-nr /proc/sys/fs/inode-state Once you've pasted this information, we'll see if we can figure this out. Sincerely, Justin L. Spies - This mail sent through IMP: http://horde.org/imp/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
