You might want to search for getHostName by itself since it's an instance method and the variable name for the object would most certainly not be InetAddress.
George Sexton MH Software, Inc. http://www.mhsoftware.com/ Voice: 303 438 9585 > -----Original Message----- > From: Carl [mailto:c...@etrak-plus.com] > Sent: Wednesday, February 03, 2010 12:55 PM > To: Tomcat Users List > Subject: Re: Tomcat dies suddenly > > George, > > I scanned the code for 'InetAddress.getHostName' and didn't find any > instances (that is what I thought but wanted to be certain.) > > It is frustrating how it will run very well for days and then suddenly > fail > every 15 minutes. Last Friday was a 'fail every 15 minutes' day. The > only > thing that was different last Friday was it was the last day of > registration > for several organizations and, while the total number of users was > still > around 80, the amount of work they were doing was substantially more > than > usual. The only server I was using that day was a Dell T105. When I > switched back to an older server (P4 dinosaur running 32 bit 5.5 java > and an > old Tomcat), it never failed. > > On the weekend, I was load testing both the rebuilt T110 and the T105 > that > was running Friday and found that the T105 had zero swap although it > has 6GB > of memory and I don't think it ever got close to that amount used (I > also > now rebuilt that server.) At any rate, no swap could, if I understand > it, > provoke the OOM killer which would give the type of failure we were > seeing. > > The load testing showed nothing. I could overload the server, run it > out of > memory, etc. and it still behaved as expected (a little weird at times > but > never the type of failure we were seeing.) > > I have now added some java options: > > -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails - > XX:+PrintGCTimeStamps > -XX:+HeapDumpOnOutOfMemoryError - > XX:HeapDumpPath=/usr/local/tomcat/logs > > and am monitoring it on a continual basis using VisualJVM (which shows > a lot > of information but has never shown anything at the time the system > fails.) > So far, the system is running great: quick response, memory in the 300- > 400MB > range, GC working as it should, plenty of free memory in the machine > (not > close to using swap) and no dying. > > If you can think of any rock I might look under, I would like to hear > about > it. > > TIA, > > Carl > > > ----- Original Message ----- > From: "George Sexton" <geor...@mhsoftware.com> > To: "'Tomcat Users List'" <users@tomcat.apache.org> > Sent: Wednesday, February 03, 2010 1:55 PM > Subject: RE: Tomcat dies suddenly > > > >> -----Original Message----- > >> From: Carl [mailto:c...@etrak-plus.com] > >> Sent: Saturday, January 30, 2010 10:32 AM > >> To: Tomcat Users List > >> Subject: Re: Tomcat dies suddenly > >> > >> George, > >> > >> Thanks for your reply. > >> > >> I will check that IPV6 is disabled... can't remember for certain. > >> > >> I have had the problem on both openSuse and Slackware which has lead > me > >> to > >> believe the problem is in the JVM. I guess I could try compiling > the > >> JVM > >> from source on the target machine. > > > > Are you doing any kind of reverse DNS lookup? E.G. > > InetAddress.getHostName()? > > > > I've really seen this problem more than once with Linux. On both 32- > bit > > and > > 64-bit systems. > > > > What makes it crazy is that the system might run for weeks, but then > a > > certain user can trigger the error and it will happen twice in an > hour. > > > > If the defect is in GLIBC, it would happen on both distributions. > Really, > > if > > you're using InetAddress.getHostName(), this could be a problem. > > > > > > > >> > >> I ran memTest86 for 30 hours on the first server I encountered the > >> problem > >> with (the first server with Slackware and 64 bit java that I brought > >> up) and > >> nothing showed up. I then, redid that machine with openSuse > (thought > >> the > >> problem might be Slackware) but openSuse failed as quickly. > >> > >> Thanks, > >> > >> Carl > >> > >> ----- Original Message ----- > >> From: "George Sexton" <geor...@mhsoftware.com> > >> To: "'Tomcat Users List'" <users@tomcat.apache.org> > >> Sent: Saturday, January 30, 2010 11:46 AM > >> Subject: RE: Tomcat dies suddenly > >> > >> > >> > I've had this happen. Finally, I got a stack trace. In my case, > there > >> > appears to be a bug in GLIBC, so when a reverse IP address lookup > is > >> done > >> > AND there is an IPV6 entry, it causes a problem. > >> > > >> > The solution appeared to be disabling IPV6 since I'm not using it. > >> This is > >> > on OpenSuSE. > >> > > >> > It would be worth checking. > >> > > >> > You might also run MemTest86+ on it if you haven't already. > >> > > >> > George Sexton > >> > MH Software, Inc. > >> > http://www.mhsoftware.com/ > >> > Voice: 303 438 9585 > >> > > >> > > >> >> -----Original Message----- > >> >> From: Carl [mailto:c...@etrak-plus.com] > >> >> Sent: Friday, January 29, 2010 12:54 PM > >> >> To: Tomcat Users List > >> >> Subject: Tomcat dies suddenly > >> >> > >> >> 6-7 weeks ago, we built up some new servers and started having > >> sudden > >> >> failures... Tomcat just stops with no error message, no system > error > >> >> messages, nothing that I have been able to find so far. > >> >> > >> >> To refresh everyone's memory, this is a new server, a Dell T110 > with > >> a > >> >> Xeon 3440 processor and 4GB memory. I have turned off both the > >> turbo > >> >> mode and hyperthreading. > >> >> > >> >> The environment: > >> >> > >> >> 64 bit Slackware Linux > >> >> > >> >> java version "1.6.0_17" > >> >> Java(TM) SE Runtime Environment (build 1.6.0_17-b04) > >> >> Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) > >> >> > >> >> Tomcat: apache-tomcat-6.0.20 > >> >> > >> >> These are the current JAVA_OPTS="-Xms1024m -Xmx1024m - > >> XX:PermSize=368m > >> >> -XX:MaxPermSize=368m" > >> >> > >> >> I have observed the memory usage and general performance with > Java > >> >> VisualVM and have seen nothing strange. I thought I was seeing > GC > >> as > >> >> memory usage was going up and down but in fact it was mostly > people > >> >> coming onto the system and leaving it. After several hours, the > >> memory > >> >> settles to a baseline of about 375MB. Forced GC never takes it > >> below > >> >> that value and the ups and downs from the people coming onto and > >> >> leaving the system also returns it to pretty much that value. > The > >> >> maximum memory used never was above 700MB for the entire day. > >> >> > >> >> The server runs well, idling along at 2-5% load, except for a > quick > >> >> spike during GC, serving jsp's, etc. at a reasonable speed. > Without > >> >> warning and with no tracks in any log (Tomcat or system) or to > the > >> >> console, Tomcat just shuts down. I can usually simply restart it > as > >> >> the ports used by Tomcat are closed... today, I needed to run > >> >> shutdown.sh before I could run startup.sh (startup.sh gave no > errors > >> >> but would not start Tomcat until I ran shutdown.sh and that > process > >> put > >> >> nothing in the logs... this is the first time this has happened.) > >> >> > >> >> Sometimes, the system will run for a week, sometimes for only > >> several > >> >> hours, sometimes only for a few minutes. Today, it ran until > about > >> >> 1:00PM and has been down four times since then. > >> >> > >> >> The failure (Tomcat shutting down) is not always the same place > in > >> the > >> >> code (I have some debugging messages going to catalina.out.) > >> >> > >> >> Load does not seem to make a difference. > >> >> > >> >> I have tried another sever (Dell T105, AMD processor, 6GB memory) > >> and > >> >> have observed the same results. I have run memTest86 on the > T110 > >> for > >> >> about 30 hours and it showed nothing. > >> >> > >> >> I rebuilt the T110 with SUSE linux, Java 1.6.18 and Tomcat > 6.0.24... > >> it > >> >> lasted 15 minutes. I have used the same server.xml on all the > >> >> installs: > >> >> > >> >> <Server port="8005" shutdown="SHUTDOWN"> > >> >> <!--APR library loader. Documentation at /docs/apr.html --> > >> >> > >> >> <Listener > className="org.apache.catalina.core.AprLifecycleListener" > >> >> SSLEngine="on" /> > >> >> > >> >> <!--Initialize Jasper prior to webapps are loaded. Documentation > at > >> >> /docs/jasper-howto.html --> > >> >> > >> >> <Listener className="org.apache.catalina.core.JasperListener" /> > >> >> > >> >> <!-- JMX Support for the Tomcat server. Documentation at > /docs/non- > >> >> existent.html --> > >> >> > >> >> <Listener > >> >> className="org.apache.catalina.mbeans.ServerLifecycleListener" /> > >> >> > >> >> <Listener > >> >> > >> > className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" > >> >> /> > >> >> > >> >> <!-- Global JNDI resources > >> >> > >> >> Documentation at /docs/jndi-resources-howto.html > >> >> > >> >> --> > >> >> > >> >> <GlobalNamingResources> > >> >> > >> >> <!-- Editable user database that can also be used by > >> >> > >> >> UserDatabaseRealm to authenticate users > >> >> > >> >> --> > >> >> > >> >> <Resource name="UserDatabase" auth="Container" > >> >> > >> >> type="org.apache.catalina.UserDatabase" > >> >> > >> >> description="User database that can be updated and saved" > >> >> > >> >> factory="org.apache.catalina.users.MemoryUserDatabaseFactory" > >> >> > >> >> pathname="conf/tomcat-users.xml" /> > >> >> > >> >> </GlobalNamingResources> > >> >> > >> >> <!-- A "Service" is a collection of one or more "Connectors" that > >> share > >> >> > >> >> a single "Container" Note: A "Service" is not itself a > "Container", > >> >> > >> >> so you may not define subcomponents such as "Valves" at this > level. > >> >> > >> >> Documentation at /docs/config/service.html > >> >> > >> >> --> > >> >> > >> >> <Service name="Catalina"> > >> >> > >> >> > >> >> <!--The connectors can use a shared executor, you can define one > or > >> >> more named thread pools--> > >> >> > >> >> <!-- > >> >> > >> >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" > >> >> > >> >> maxThreads="150" minSpareThreads="4"/> > >> >> > >> >> --> > >> >> > >> >> > >> >> > >> >> <!-- A "Connector" represents an endpoint by which requests are > >> >> received > >> >> > >> >> and responses are returned. Documentation at : > >> >> > >> >> Java HTTP Connector: /docs/config/http.html (blocking & non- > >> blocking) > >> >> > >> >> Java AJP Connector: /docs/config/ajp.html > >> >> > >> >> APR (HTTP/AJP) Connector: /docs/apr.html > >> >> > >> >> Define a non-SSL HTTP/1.1 Connector on port 8080 > >> >> > >> >> --> > >> >> > >> >> <Connector port="8080" protocol="HTTP/1.1" > >> >> > >> >> maxHttpHeaderSize="8192" > >> >> > >> >> maxThreads="600" minSpareThreads="25" maxSpareThreads="75" > >> >> > >> >> enableLookups="false" redirectPort="8443" scheme="http" > >> >> acceptCount="100" > >> >> > >> >> connectionTimeout="20000" disableUploadTimeout="true" /> > >> >> > >> >> <!-- A "Connector" using the shared thread pool--> > >> >> > >> >> <!-- > >> >> > >> >> <Connector executor="tomcatThreadPool" > >> >> > >> >> port="8080" protocol="HTTP/1.1" > >> >> > >> >> connectionTimeout="20000" > >> >> > >> >> redirectPort="8443" /> > >> >> > >> >> --> > >> >> > >> >> <!-- Define a SSL HTTP/1.1 Connector on port 8443 > >> >> > >> >> This connector uses the JSSE configuration, when using APR, the > >> >> > >> >> connector should be using the OpenSSL style configuration > >> >> > >> >> described in the APR documentation --> > >> >> > >> >> > >> >> <Connector port="8443" maxHttpHeaderSize="8192" > >> >> > >> >> maxThreads="600" minSpareThreads="25" maxSpareThreads="75" > >> >> > >> >> enableLookups="false" disableUploadTimeout="true" > >> >> > >> >> acceptCount="100" scheme="https" secure="true" > >> >> > >> >> clientAuth="false" sslProtocol="TLS" SSLEnabled="true" > >> >> > >> >> keystoreFile="/usr/local/certs/tomcat_keystore.ks" > >> >> keystorePass="jellybean"/> > >> >> > >> >> > >> >> <!-- Define a SSL HTTP/1.1 Connector on port 443 --> > >> >> > >> >> > >> >> <Connector port="443" maxHttpHeaderSize="8192" > >> >> > >> >> maxThreads="600" minSpareThreads="25" maxSpareThreads="75" > >> >> > >> >> enableLookups="false" disableUploadTimeout="true" > >> >> > >> >> acceptCount="100" scheme="https" secure="true" > >> >> > >> >> clientAuth="false" sslProtocol="TLS" SSLEnabled="true" > >> >> > >> >> keystoreFile="/usr/local/certs/tomcat_keystore.ks" > >> >> keystorePass="jellybean"/> > >> >> > >> >> > >> >> <!-- > >> >> > >> >> <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" > >> >> > >> >> maxThreads="150" scheme="https" secure="true" > >> >> > >> >> clientAuth="false" sslProtocol="TLS" /> > >> >> > >> >> --> > >> >> > >> >> <!-- Define an AJP 1.3 Connector on port 8009 --> > >> >> > >> >> <Connector port="8009" > >> >> > >> >> enableLookups="false" redirectPort="443" protocol="AJP/1.3" /> > >> >> > >> >> > >> >> > >> >> <!-- An Engine represents the entry point (within Catalina) that > >> >> processes > >> >> > >> >> every request. The Engine implementation for Tomcat stand alone > >> >> > >> >> analyzes the HTTP headers included with the request, and passes > them > >> >> > >> >> on to the appropriate Host (virtual host). > >> >> > >> >> Documentation at /docs/config/engine.html --> > >> >> > >> >> <!-- You should set jvmRoute to support load-balancing via AJP ie > : > >> >> > >> >> <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1"> > >> >> > >> >> --> > >> >> > >> >> <Engine name="Catalina" defaultHost="localhost"> > >> >> > >> >> <!--For clustering, please take a look at documentation at: > >> >> > >> >> /docs/cluster-howto.html (simple how to) > >> >> > >> >> /docs/config/cluster.html (reference documentation) --> > >> >> > >> >> <!-- > >> >> > >> >> <Cluster > className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/> > >> >> > >> >> --> > >> >> > >> >> <!-- The request dumper valve dumps useful debugging information > >> about > >> >> > >> >> the request and response data received and sent by Tomcat. > >> >> > >> >> Documentation at: /docs/config/valve.html --> > >> >> > >> >> <!-- > >> >> > >> >> <Valve > className="org.apache.catalina.valves.RequestDumperValve"/> > >> >> > >> >> --> > >> >> > >> >> <!-- This Realm uses the UserDatabase configured in the global > JNDI > >> >> > >> >> resources under the key "UserDatabase". Any edits > >> >> > >> >> that are performed against this UserDatabase are immediately > >> >> > >> >> available for use by the Realm. --> > >> >> > >> >> <Realm className="org.apache.catalina.realm.UserDatabaseRealm" > >> >> > >> >> resourceName="UserDatabase"/> > >> >> > >> >> <!-- Define the default virtual host > >> >> > >> >> Note: XML Schema validation will not work with Xerces 2.2. > >> >> > >> >> --> > >> >> > >> >> <Host name="localhost" appBase="webapps" > >> >> > >> >> unpackWARs="true" autoDeploy="true" deployOnStartup="true" > >> >> > >> >> xmlValidation="false" xmlNamespaceAware="false"> > >> >> > >> >> <!-- SingleSignOn valve, share authentication between web > >> applications > >> >> > >> >> Documentation at: /docs/config/valve.html --> > >> >> > >> >> <!-- > >> >> > >> >> <Valve className="org.apache.catalina.authenticator.SingleSignOn" > /> > >> >> > >> >> --> > >> >> > >> >> <!-- Access log processes all example. > >> >> > >> >> Documentation at: /docs/config/valve.html --> > >> >> > >> >> <!-- > >> >> > >> >> <Valve className="org.apache.catalina.valves.AccessLogValve" > >> >> directory="logs" > >> >> > >> >> prefix="localhost_access_log." suffix=".txt" pattern="common" > >> >> resolveHosts="false"/> > >> >> > >> >> --> > >> >> > >> >> </Host> > >> >> > >> >> </Engine> > >> >> > >> >> </Service> > >> >> > >> >> </Server> > >> >> > >> >> When Tomcat shuts down, the memory that it was using seems to > still > >> be > >> >> held (as seen from top) but it is nowhere near the machine > physical > >> >> memory. > >> >> > >> >> The application has been running on an older server (Dell 600SC, > 32 > >> bit > >> >> Slackware, 2GB memory) for several years and, while the > application > >> >> will throw exceptions now and then, it never crashed. This lead > me > >> to > >> >> believe the problem had something to do with the 64 bit JVM but, > >> with > >> >> without seeing errors anywhere, I can't be certain and don't know > >> what > >> >> I can do about it except go back to 32 bit. > >> >> > >> >> One time, I observed the heap and permGen memory usage with > Visual > >> JVM. > >> >> It was running around 600MB before I forced a GC and 375MB > >> afterward. > >> >> Speed was good. Memory usage from top was 2.4GB. Five minutes > >> later, > >> >> Tomcat stopped leaving no tracks that I could find. The memory > >> usage > >> >> from top was around 2.4GB. The memory usage from Visual JVM was > >> still > >> >> showing 400MB+ although the Tomcat process was gone. I restarted > >> >> Tomcat (did not reboot) so Tomcat had been shutdown gracefully > >> enough > >> >> to close the ports (8080, 8443, 443.) Tomcat stayed up for less > >> than > >> >> an hour (under light load) and stopped again. The memory used > >> >> according to top was less than 3GB but I didn't get the exact > >> number. > >> >> I restarted it again (no server reboot) and it ran for the rest > of > >> the > >> >> night (light load) and top was showing 3.3GB for memory in the > >> morning. > >> >> > >> >> Anyone have any ideas how I might track this problem down? > >> >> > >> >> Thanks, > >> >> > >> >> Carl > >> > > >> > > >> > ------------------------------------------------------------------ > --- > >> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> > For additional commands, e-mail: users-h...@tomcat.apache.org > >> > > >> > > >> > >> > >> -------------------------------------------------------------------- > - > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
