FtpServer hitting its limit?
Hi. I have an issue with apache FtpServer. It's set to use passive mode, and it looks like all of the ports are being used, which stops any new connections from being connected and the whole ftp server just hangs. If you try to connect it just reaches time out. Here what I get after this happens, in the log: [ INFO] 2009-10-08 09:04:44,347 [] [65.75.250.58] CREATED [ INFO] 2009-10-08 09:04:44,753 [] [64.90.174.226] CREATED [ INFO] 2009-10-08 09:04:45,284 [] [72.89.239.125] CREATED [ INFO] 2009-10-08 09:04:45,909 [] [195.14.162.97] CREATED [ INFO] 2009-10-08 09:04:47,018 [] [85.225.141.116] CREATED [ INFO] 2009-10-08 09:04:47,909 [] [38.99.250.2] CREATED [ INFO] 2009-10-08 09:04:48,268 [] [114.111.139.57] CREATED [ INFO] 2009-10-08 09:04:48,972 [] [80.34.69.143] CREATED [ INFO] 2009-10-08 09:04:49,456 [] [111.67.83.18] CREATED [ INFO] 2009-10-08 09:04:49,472 [] [202.150.209.90] CREATED [ INFO] 2009-10-08 09:04:49,472 [] [118.243.143.178] CREATED [ INFO] 2009-10-08 09:04:49,550 [] [81.247.77.182] CREATED [ INFO] 2009-10-08 09:04:49,768 [] [220.236.247.154] CREATED [ INFO] 2009-10-08 09:04:50,347 [] [79.172.211.17] CREATED [ INFO] 2009-10-08 09:04:50,550 [] [174.133.32.49] CREATED [ INFO] 2009-10-08 09:04:50,565 [] [81.247.77.182] CREATED [ INFO] 2009-10-08 09:04:51,612 [] [75.127.87.129] CREATED And it's just keeps going on like that. I have about 1000 connections every 5 minutes which creates about 35gb of daily incoming traffic, so I think I'm hitting the limit of the apache FtpServer. How can I avoid it? Is there a workaround? Also, since only one connection is allowed per port, is there a way to virtually increase the number of ports to allow more simultaneous connections? Thanks.
Re: FtpServer hitting its limit?
On Thu, Oct 8, 2009 at 11:39 AM, a...@safe-mail.net wrote: I have about 1000 connections every 5 minutes which creates about 35gb of daily incoming traffic, so I think I'm hitting the limit of the apache FtpServer. How can I avoid it? Is there a workaround? Interesting workload. However, I do not know of any limitations that should cause this. Also, since only one connection is allowed per port, is there a way to virtually increase the number of ports to allow more simultaneous connections? Have you restricted the ports FtpServer is allowed to use for passive connections? Also, could it be that you use up the file handles (sockets) for your OS? /niklas
Re: FtpServer hitting its limit?
On Thu, Oct 8, 2009 at 12:49 PM, a...@safe-mail.net wrote: Not sure regarding the file handles - how do I check it? I'm using windows server 2003. I'm not too familliar with Windows 2003, but you can probably google around on max file handles. By the way, every time it happens, I just restart the ftp server and it starts working normally again. Since the ftp connections increase daily (more users), and it happens more regularly, that's why I suspect I'm hitting limit of apache FtpServer. Could you do a netstat -a when this happens so we can look at the state of sockets? /niklas
Re: FtpServer hitting its limit?
Just googled it and it says: file server running Windows Server 2003 with 1 GB of RAM can efficiently support approximately 100,000 remote concurrent file handles, regardless of the size of the files. If your users are likely to have more than 100,000 files open at a time, plan to split this load across two or more servers. So it's not likely it's the cause, as there is only 1 file per user, and there is less than 2000 users. Also as the ftp server runs fine when restarted, and it takes couple of days to reach this issue, I don't think it's the os causing this. niklas, can you give a private email to send the netstat report to(when I'll have it)? As I do not want to make it public. Thank you. On Thu, Oct 8, 2009 at 12:49 PM, a...@safe-mail.net wrote: Not sure regarding the file handles - how do I check it? I'm using windows server 2003. I'm not too familliar with Windows 2003, but you can probably google around on max file handles. By the way, every time it happens, I just restart the ftp server and it starts working normally again. Since the ftp connections increase daily (more users), and it happens more regularly, that's why I suspect I'm hitting limit of apache FtpServer. Could you do a netstat -a when this happens so we can look at the state of sockets? /niklas
Re: FtpServer hitting its limit?
On Thu, Oct 8, 2009 at 1:46 PM, a...@safe-mail.net wrote: Just googled it and it says: file server running Windows Server 2003 with 1 GB of RAM can efficiently support approximately 100,000 remote concurrent file handles, regardless of the size of the files. If your users are likely to have more than 100,000 files open at a time, plan to split this load across two or more servers. So it's not likely it's the cause, as there is only 1 file per user, and there is less than 2000 users. The problem with file handles is not due to the file the user downloads, but the sockets used by the server. Each socket consumes a file handle, and the socket lives also after the user has disconnect for a certain time. If too many users connect during this period, you might run out of handles. The max allowed handles is a setting in the OS. You can find some more info in our FAQ: http://mina.apache.org/ftpserver/faq.html Also as the ftp server runs fine when restarted, and it takes couple of days to reach this issue, I don't think it's the os causing this. I agree this points to some other issue, but let make sure. niklas, can you give a private email to send the netstat report to(when I'll have it)? As I do not want to make it public. Send it to the address from which I send these emails. Also, can you include a thread dump? /niklas
Re: FtpServer hitting its limit?
If it is an issue with Socket close timeouts, can we have all sockets created by the FtpServer to set SO_LINGER to zero or a small number? Basically, call Socket.setSolinger on every socket we/MINA creates. Do you think this is the same setting as the one that you pointed in the FAQ? Regards, Sai Pullabhotla www.jMethods.com On Thu, Oct 8, 2009 at 6:53 AM, Niklas Gustavsson nik...@protocol7.com wrote: On Thu, Oct 8, 2009 at 1:46 PM, a...@safe-mail.net wrote: Just googled it and it says: file server running Windows Server 2003 with 1 GB of RAM can efficiently support approximately 100,000 remote concurrent file handles, regardless of the size of the files. If your users are likely to have more than 100,000 files open at a time, plan to split this load across two or more servers. So it's not likely it's the cause, as there is only 1 file per user, and there is less than 2000 users. The problem with file handles is not due to the file the user downloads, but the sockets used by the server. Each socket consumes a file handle, and the socket lives also after the user has disconnect for a certain time. If too many users connect during this period, you might run out of handles. The max allowed handles is a setting in the OS. You can find some more info in our FAQ: http://mina.apache.org/ftpserver/faq.html Also as the ftp server runs fine when restarted, and it takes couple of days to reach this issue, I don't think it's the os causing this. I agree this points to some other issue, but let make sure. niklas, can you give a private email to send the netstat report to(when I'll have it)? As I do not want to make it public. Send it to the address from which I send these emails. Also, can you include a thread dump? /niklas
Re: FtpServer hitting its limit?
I think the SO_LINGER is related to the time it hangs around even though when the socket is closed. I also see this as a configurable option in Tomcat's connector configuration. The tomcat's documentation describes this field as below: connectionLinger The number of milliseconds during which the sockets used by this Connector will linger when they are closed. The default value is -1 (socket linger is disabled). If we disable lingering of sockets, it may avoid issues of too many open handles. However, there may be some packet losses? I'm no expert in TCP/IP, but I'm hoping some one from the MINA team could shed more light on this. Do you think we should start a thread on the developers list? Sai Pullabhotla www.jMethods.com On Thu, Oct 8, 2009 at 8:05 AM, Niklas Gustavsson nik...@protocol7.com wrote: On Thu, Oct 8, 2009 at 2:28 PM, Sai Pullabhotla sai.pullabho...@jmethods.com wrote: If it is an issue with Socket close timeouts, can we have all sockets created by the FtpServer to set SO_LINGER to zero or a small number? First, I'm not sure we should do that. Second, I'm not sure if SO_LINGER affects the entire socket shutdown process so I'm not sure it would even help in this case. /niklas
Re: FtpServer hitting its limit?
There is a good article on the time_wait state of sockets at http://www.developerweb.net/forum/showthread.php?t=2941 which mentions the SO_LINGER settings: ...Because of these potential problems with TIME_WAIT assassinations, one should not avoid the TIME_WAIT state by setting the SO_LINGER option to send an RST instead of the normal TCP connection termination (FIN/ACK/FIN/ACK). The TIME_WAIT state is there for a reason; it's your friend and it's there to help you :-) Might be of some help. Best Regards, Gary
