Re: Stability and reliability issues
Were you able to get a netstat -a or lsof output from the hung server? /niklas On Fri, Apr 10, 2009 at 6:05 PM, Dan d...@tipjarawards.com wrote: Versions we are using: RC1 and Mina M4 inside of tomcat 5.5.x on linux 32-bit. Another crash after two days or working, had to restart server. So in a way this is good news, worse would be crashes every month, I'm sure we can track this bug down. We are running the server on its own machine so it is running in a very isolated environment. After the crash the server stays running to some extent but nobody can connect (endless CREATED log messages but no logins), here is the Filezilla output: Status: Connecting to 174.129.220.134:21... Status: Connection established, waiting for welcome message... Error: Connection timed out Error: Could not connect to server Can't give a DEBUG log output since simply too much output from other components that are running. Here is the last snippet of INFO level logging before crash: Apr 10, 2009 3:32:12 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: SENT: 250 Directory changed to /XXX/files/SyncData^M Apr 10, 2009 3:32:12 PM org.apache.ftpserver.listener.nio.FtpLoggingFilter messageReceived INFO: RECEIVED: MODE S Apr 10, 2009 3:32:12 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: SENT: 200 Command MODE okay.^M Apr 10, 2009 3:32:13 PM org.apache.ftpserver.listener.nio.FtpLoggingFilter messageReceived INFO: RECEIVED: STRU F Apr 10, 2009 3:32:13 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: SENT: 200 Command STRU okay.^M Apr 10, 2009 3:32:13 PM org.apache.ftpserver.listener.nio.FtpLoggingFilter messageReceived INFO: RECEIVED: TYPE I Apr 10, 2009 3:32:13 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: SENT: 200 Command TYPE okay.^M Apr 10, 2009 3:32:13 PM org.apache.ftpserver.listener.nio.FtpLoggingFilter messageReceived INFO: RECEIVED: PASV Apr 10, 2009 3:33:15 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: CREATED Apr 10, 2009 3:33:35 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: CREATED Apr 10, 2009 3:33:55 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: CREATED Apr 10, 2009 3:34:15 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: CREATED Apr 10, 2009 3:34:35 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: CREATED Apr 10, 2009 3:34:55 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: CREATED Apr 10, 2009 3:35:15 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: CREATED Apr 10, 2009 3:35:37 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: CREATED Apr 10, 2009 3:35:57 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: CREATED Apr 10, 2009 3:36:17 PM org.apache.mina.filter.logging.LoggingFilter log WARNING: CREATED -Original Message- From: David Latorre [mailto:dvl...@gmail.com] Sent: April-07-09 3:44 AM To: ftpserver-users@mina.apache.org Subject: Re: Stability and reliability issues My answers are below: 2009/4/6 Dan d...@tipjarawards.com: We are using RC1 with Mina M4 and we start the server embedded in a tomcat 5.5.x webapp. Java is version 5.0 running on Linux. There was another user with the same Max login problem as I previously posted that problem to this group. Even when the ftp server dies the tomcat server is running fine throughout, still taking connections etc. Once we reboot the ftp server starts working again. Can you provide more info on your environment? For example: are you updated to the latest 5.0 version? Are you using SSL or plain connections )? Active or Passive? both of them? Since you're integrating FtpServer into Tomcat I do not know which logging facility you're using: is it log4j? In that case, DEBUG level is set in log4j.properties; the ready-to-use ftpserver bundled has a log4j.properties, but i'd rather use a file appender. At this point my guess is there is somewhere in the login workflow where something isn't being closed properly. We have several users running batch scripts, if they aren't closing a connection etc. that should not be able to crash the ftp server. Maybe connections are not being closed or collected with an idle timeout in some cases? Once you set the LOG level to debug can you send it along with the configuration you are using (you aren't using the xml config unfortunately). Are you using custom Ftplets? For now we are just going to restart the server and wait and see. Thanks, Dan -Original Message- From: West, John C [mailto:john.c.w...@citizensbank.com] Sent: April-06-09 2:00 PM To: ftpserver-users@mina.apache.org Subject: RE: Stability and reliability issues Well I'm using this server
Re: Stability and reliability issues
I have seen similar ocurrences to the cannot connect to server incident mentioned. For us, this was caused by the ftpserver _sometimes_ picking a (passive) port that our firewall was not allowing through from the client. I changed the passive port range setting in ftpserver to match the firewall port range and it has worked ok since. I guess it therefore follows you could also get this if you have set up a port range and you run out of ports momentarily under heavy demand. This can also happen with clients such as filezilla that can open many ports at once to transfer files. I have also seen the max logins reached error as well for a user account that was just set up and therefore could not have reached any limit. Since this was the first occurrence in many months of continuous operation, I just restarted the server and everything was OK. I might peer more closely into the source code now. Best Regards, Gary Bell
RE: Stability and reliability issues
Well I'm using this server in a couple different production environments without any issues and I know there are others on this list who are as well, so I'd say your experience is not typical. What version of the ftp server and Java are you using? What operating system? The debug level is set in log4j.properties. -Original Message- From: Dan [mailto:d...@tipjarawards.com] Sent: Monday, April 06, 2009 1:51 PM To: ftpserver-users@mina.apache.org Subject: Stability and reliability issues We are having some serious stability/scalability issues with the ftp server. First there were hanging logins (nobody able to login but supposedly max login users was reacehd according to log message). With idle timeouts this should never happen as after a few minutes all the dead logins should have been collected. Second problem we are now getting 'Could not connect to server' and 'Connection timed out' errors with the server running after only two days. It gets as far as the CREATED log message then hangs as a user tries to connect and login. So we will be rebooting the server again as it appears the server is again dead. I hope the focus of development is first and foremost stability/reliability since adding in new features to a server that keeps crashing in a production environment is very troubling. I am happy to provide more debug info or log info to help, it's just not clear how to accomplish this. Where can I easily set the debug level? Any ideas? Let me know. Thanks, Dan - Use of email is inherently insecure. Confidential information, including account information, and personally identifiable information, should not be transmitted via email, or email attachment. In no event shall Citizens or any of its affiliates accept any responsibility for the loss, use or misuse of any information including confidential information, which is sent to Citizens or its affiliates via email, or email attachment. Citizens does not guarantee the accuracy of any email or email attachment, that an email will be received by Citizens or that Citizens will respond to any email. This email message is confidential and/or privileged. It is to be used by the intended recipient only. Use of the information contained in this email by anyone other than the intended recipient is strictly prohibited. If you have received this message in error, please notify the sender immediately and promptly destroy any record of this email.
RE: Stability and reliability issues
The good news is we are now running a test server so we should have a clue next time an issue arises. Dan -Original Message- From: West, John C [mailto:john.c.w...@citizensbank.com] Sent: April-06-09 2:00 PM To: ftpserver-users@mina.apache.org Subject: RE: Stability and reliability issues Well I'm using this server in a couple different production environments without any issues and I know there are others on this list who are as well, so I'd say your experience is not typical. What version of the ftp server and Java are you using? What operating system? The debug level is set in log4j.properties. -Original Message- From: Dan [mailto:d...@tipjarawards.com] Sent: Monday, April 06, 2009 1:51 PM To: ftpserver-users@mina.apache.org Subject: Stability and reliability issues We are having some serious stability/scalability issues with the ftp server. First there were hanging logins (nobody able to login but supposedly max login users was reacehd according to log message). With idle timeouts this should never happen as after a few minutes all the dead logins should have been collected. Second problem we are now getting 'Could not connect to server' and 'Connection timed out' errors with the server running after only two days. It gets as far as the CREATED log message then hangs as a user tries to connect and login. So we will be rebooting the server again as it appears the server is again dead. I hope the focus of development is first and foremost stability/reliability since adding in new features to a server that keeps crashing in a production environment is very troubling. I am happy to provide more debug info or log info to help, it's just not clear how to accomplish this. Where can I easily set the debug level? Any ideas? Let me know. Thanks, Dan - Use of email is inherently insecure. Confidential information, including account information, and personally identifiable information, should not be transmitted via email, or email attachment. In no event shall Citizens or any of its affiliates accept any responsibility for the loss, use or misuse of any information including confidential information, which is sent to Citizens or its affiliates via email, or email attachment. Citizens does not guarantee the accuracy of any email or email attachment, that an email will be received by Citizens or that Citizens will respond to any email. This email message is confidential and/or privileged. It is to be used by the intended recipient only. Use of the information contained in this email by anyone other than the intended recipient is strictly prohibited. If you have received this message in error, please notify the sender immediately and promptly destroy any record of this email.
