RE: RE : Tomcat 6.0.35 Crashed again
> From: saumil shah [mailto:saumil...@hotmail.com] > Subject: RE : Tomcat 6.0.35 Crashed again > It would be "prefered" to make Min and Max heap size same Usually that is the case. > is -XX:+UseParallelGC recommended , since I have 8 core machine or would > there be penalty to that ? You very likely should use one of the parallel collector settings. Depending on the JVM version and the platform you're running on, it may already be enabled by default. You might want to review this: http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html Although written for Java 6, it's still applicable to Java 7 and 8. There is a newer parallel and concurrent collector (G1) available in Java 7, but I'm not sure what it's stability is like yet. It is destined to be the default collector one of these days. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE : Tomcat 6.0.35 Crashed again
Hello All, First of all ...Thank you so very much Daniel, Neven, Marc, Howard, Christopher.appreciate all the help. Learnt a lot. What I gather from the chain is that, It would be "prefered" to make Min and Max heap size same . i.e. in my case 4096m vs Min as 1024m and max 4096m which may waste some memory if NOT required, but for now , we have ample free memory.It appears, Viewing via VisualVM or JConsole even for local tomcat install , for windows service installer , does require JMX port setting )(http://stackoverflow.com/questions/503057/unable-to-use-jconsole-with-tomcat-running-as-windows-service) and hence restart is a must ?Other question was , is -XX:+UseParallelGC recommended , since I have 8 core machine or would there be penalty to that ? Once again many thanks and appreciate all the help.
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Chris, My apologies for late response; just realized earlier this afternoon that I didn't respond. On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Howard, > > On 4/3/13 4:15 PM, Howard W. Smith, Jr. wrote: > > On Tue, Apr 2, 2013 at 5:12 PM, Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > >> If you don't re-deploy your webapp, then daily rolling Tomcat > >> restarts are not necessary. I wonder why you are re-deploying > >> your web application so many times? > > > > As a new tomcat user and still somewhat junior java/jsf developer, > > I restart tomcat whenever I have new software changes to > > deploy-and-want-to-run-on the production server. sometimes, I > > deploy-and-restart multiple times per day, but sometimes, I'm able > > to let tomcat/tomee run for days without restart. > > That's not really conducive to high-availability. Are you using > Tomcat's parallel-deployment feature? > Agreed, and not using parallel-deployment feature at the moment. > > >> We run several Tomcats in parallel with modest heaps (less than > >> 512MiB each) and they can run for months before we stop them for > >> upgrades. It *is* possible to run JVMs without running out of > >> memory... > > > > > > I too, have not experienced any OOME, and recently, per what I > > have seen-and-read of other (more senior java developers than > > myself), I have decreased memory settings in my java options on > > tomcat7w.exe (see below). > > > > -Xmx1024m -XX:MaxPermSize=384m -XX:+UseTLAB > > -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled > > Your heap settings should be tailored to your environment and usage > scenarios. Interesting. I suppose 'your environment' means memory available, operating system, hardware. Usage scenarios? hmmm... please clarify with a brief example, thanks. :) heap settings tailored to 'my' environment and usage... hmmm, not many users hitting the app, app is not used 'all day long', app has @Schedule tasks that connects to an email acct, downloads customer email requests, and inserts customer requests into database (Martin recommended to close resources; sometime ago, I had to refactor all of that code, and I am closing the connection to the email acct, and open the connection when @Schedule tasks are executed), i am using JMS via TomEE/activeMQ to perform some tasks, asynchronously (tomee committers told me that use of @Asynchronous would be better, and less overhead); honestly, I do open 2 or 3 JMS resources/queues in @ApplicationScoped @PostConstruct (if I'm not mistaking) and close those resources in @ApplicationScoped @PreDestroy; why? I think I read on ActiveMQ site/documentation, where they recommend that that is better on performance, than open/close-on-demand. Almost forgot...as I mentioned in another thread, as enduser changes data, I have an implementation that keeps google calendar in sync with the database, which involves JMS/ActiveMQ/MDB and many/multiple requests to google calendar API. hmmm, more about usage, I have the following: JdbcDriver org.apache.derby.jdbc.EmbeddedDriver JdbcUrl jdbc:derby:;create=true UserName Password JtaManaged true jmxEnabled true InitialSize 2 MaxActive 80 MaxIdle 20 MaxWait 1 minIdle 10 suspectTimeout 60 removeAbandoned true removeAbandonedTimeout 180 timeBetweenEvictionRunsMillis 3 jdbcInterceptors=StatementCache(max=128) > You can find "conventional wisdom" that recommends pretty > much any heap configuration you want. The only thing that I can > consistently recommend to anyone is to set -Xms and -Xmx to the same > value, since on a server you're pretty much guaranteed to get to -Xmx > pretty quickly, anwyay. You may as well not thrash the heap space(s) > getting there. > Interesting, I did set -Xms and -Xmx to the same value (as you and others-on-this-list have recommended, thanks). > > > My Windows 2008 R2 Server (64bit 32GB RAM) never seems to get > > higher than 1% CPU, and I think I do have memory leaks somewhere in > > the app, but FWIW (in heap dump in java visual vm), the memory > > leaks seem to be tomee leaks. In Java Visual VM, I do see the > > memory grow over time, as the app is used (without a tomcat restart > > or re-deploy of app and then restart tomcat), but I still have not > > seen OOME...'yet'. > > What does your heap usage graph look like? It should be a nice > sawtooth-looking thing, like this: > > /|/|/|/|/|/|/|/|/| > > I do occasionally see the sawtooth-looking graph, > You'll see that the small sawtooth pattern grows in basis over time > and then there is a major GC which will reset you back to some > baseline, then the process starts over again. > and eventually, I see the graph even out (non-sawtooth-looking graph). > > If you never get OOMEs, why do you think you have memory leaks? remember, I do restart tomee quite often, especially
Re: RE : Tomcat 6.0.35 Crashed again
On Thu, Apr 11, 2013 at 8:02 PM, Martin Gainty wrote: > you need to do take a look at the loaded JSF webapps and find outwho is > acquiring a resource and not closing the resource > who is acquiring large amounts of heap and not releasingbe aware any > reference to an any object in another class gives the class the right to be > placed into PermGenHibernate with cglib proxies are notorious memory hogs > awatch your PermGen get pegged when Hibernate and cglib proxies are > loadedStatics are another set of culprits of of heap usage > Remember all long lived heap objects are eventually placed into Permgen > Find the tools to track eden heap, tenured heap and PermGen > > http://www.integratingstuff.com/2011/07/24/understanding-and-avoiding-the-java-permgen-space-error/get > familiar with taking heap dumps with jmap and analyzing with jhathttp:// > javarevisited.blogspot.com/2011/05/java-heap-space-memory-size-jvm.htmlMartin > > Interesting. Thanks! > __ > Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité > > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene > Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte > Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht > dient lediglich dem Austausch von Informationen und entfaltet keine > rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von > E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. > Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le > destinataire prévu, nous te demandons avec bonté que pour satisfaire > informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie > de ceci est interdite. Ce message sert à l'information seulement et n'aura > pas n'importe quel effet légalement obligatoire. Étant donné que les email > peuvent facilement être sujets à la manipulation, nous ne pouvons accepter > aucune responsabilité pour le contenu fourni. > > > Date: Thu, 11 Apr 2013 11:40:46 -0400 > > Subject: Re: RE : Tomcat 6.0.35 Crashed again > > From: smithh032...@gmail.com > > To: users@tomcat.apache.org > > > > On Thu, Apr 11, 2013 at 10:41 AM, Mark H. Wood wrote: > > > > > Really, no one else can tell you what settings to use. The best we > > > can hope for is some accepted rules of thumb *as starting points* for > > > further tuning. > > > > > > > > +1 to Dan, Neven, and Mark's responses. Please consider-or-do > 'everything' > > that they mentioned/recommended. > > > > I did want to share my java settings for my > > currently-considered-a-low-scale JSF web app running on Windows Server > 2008 > > R2 64bit server with 32GB RAM. > > > > -XX:HeapDumpPath=D:\apache-tomee-plus-1.6.0-SNAPSHOT\temp > > -XX:+HeapDumpOnOutOfMemoryError > > -Djava.awt.headless=true > > -Dcom.sun.management.jmxremote.port=422 > > -Dcom.sun.management.jmxremote.ssl=false > > -Dcom.sun.management.jmxremote.authenticate=false > > -Xms1024m > > -Xmx1024m > > -XX:MaxPermSize=384m > > -XX:+UseTLAB > > -XX:+UseConcMarkSweepGC > > -XX:+CMSClassUnloadingEnabled > > > > I am very pleased with the GC performance of my app, and I do like to > > monitor the performance of the app via JMX remote connection via Java > > Visual VM. My app runs between 200m to 500m, but I am keeping > Xms/Xmx=1024m > > just to see if I ever get an OOME; so far, so good (never experienced an > > OOME), but recently, I did experience some unexpected/unwanted behavior > > with one of my @Schedule processes which was attempting to sync some data > > from database to/with Google Calendar, and google Calendar service > returned > > google calendar error 503, and I recognized that the memory got up to > 500m, > > and the google calendar error 503 did not resolve itself over an hour > > (@Schedule executes every 2 to 4 minutes, if error occurs, then data is > > appended to the queue for later retry attempt). I never seen that > behavior > > and I don't know if I will see it again; i wish I would have done a 'heap > > dump' instead of a 'stop' tomee/tomcat. Everyday, I listen and read these > > questions/responses on tomcat list, and I can't believe that I forgot to > do > > a 'heap dump'. :( > > > > Also, please note that I occasionally stop-deploy-and-start tomee/tomcat > > almost-on-a-daily-basis to deploy new app-or-configuration-or-library > > updates; also, the app or tomee (or tomcat) seem to accumlate > threadlocals > > over time, and if uptime is > 1 day, then I 'think' I see that memory is > > not released, and I think eventually as uptime increases, then the > > app/tomee/tomcat will result in OOME. :) > > > > At any rate, hope this helps. > > > > Howard > >
RE: Tomcat access log reveals hack attempt: "HEAD /manager/html HTTP/1.0" 404
> You would have had to intentionally enable the "default" password. I had clearly done that. > The attacker installed a viral servlet application that killed the > server completely, we had to rebuild it. I -- like most people I would guess -- don't run under a SecurityManager, but doing so can significantly limit the damage that a rogue webapp can do. Thanks, I don't think that's feasible given our webapps. > Did you also remove manager webapp access through httpd? Otherwise, this doesn't actually do anything to help. No I didn't, I *access-controlled it,* as I went on to describe. > +2 -- both good ideas: central access control (LDAP) and enabling > lockout mechanism. Note that Tomcat's lockout mechanism is fairly primitive and easy to game. I referred to the OpenLDAP lockout mechanism, which is not at all primitive. > Would you be willing to review the Tomcat documentation on "securing Tomcat" and make a few comments? It could always use some additional tips: Sure, will do. http://tomcat.apache.org/tomcat-7.0-doc/security-howto.html http://wiki.apache.org/tomcat/FAQ/Security You can sign-up for the wiki yourself and make any changes you want. If you want to modify the "official" documentation, create a Bugzilla enhancement request and (please!) include a patch. I'm sure it will go right in. Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRZx7AAAoJEBzwKT+lPKRY+3MP/3c8kZZU43cMaxkXTi/ELXha Hv6rAeGz4nnpMNn2C002cTRgZ39vomUXYdsLMnNIshn05JDVIZmLLoUXk6UzY9go uH0QdAubBxhvwC/CWeLjUuSjy/Ei4vKeB7xJNw/FQ2xXEt47FWv36e0vgxOyluX+ gbkB3KQlN6PXtQENGvkOGT5oWLK9M7WUydGSWq9lXR+akwWeL3jWRAlLl6bHYybQ n70c5wq/rJbEj+k9yCHsMZvPabYs5ejsz6wHvvw4Emrxcp4LVVjCuY2Z87Yhdtb4 B43tF48be9GUZCXDvtIjiS5phHMhpqyJakHuZ7GSvzDIeuiNZ96XuoDkIG1bwWjf Z5SMCSjkSPqDKJ1cXcd8AaSYgI2C3KQnuFrbTD7bVqQHOeq7RJZp3+xE0IUNPl+V H2PNpUfXD9BDbPiiDt8bcgvcrImejW0RDumQ2fwbTVvt4OaybVsMUsVFW8lUtw3A YhvFn/WCEdR8VaY9PkqYm84BVMsQJBbBdb5clYiAtVQRky1NPS+hcIihnf85DkNU vr6rv/oK0aMXAamwUagmRe5OjTHuHczERPYgEUMpppnlXuNV1mLxBib8+HInGg3/ Y5i28tTd7fS5uo7/CZv+9uEZdDUO7utWGT0W+gBaIkh35/yZI5a1l5wi0szYduQe t3nftQXUTCYtK1QNwKND =3s6Q -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Better SSL connector setup
Enough already guys please. EJP -Original Message- From: users-digest-h...@tomcat.apache.org [mailto:users-digest-h...@tomcat.apache.org] Sent: Thursday, 11 April 2013 11:48 PM To: users@tomcat.apache.org Subject: users Digest 11 Apr 2013 13:48:25 - Issue 11342 users Digest 11 Apr 2013 13:48:25 - Issue 11342 Topics (messages 241110 through 241119) Re: Better SSL connector setup 241110 by: Mark Eggers Re: Resource management in new Tomcat JDBC connection pool. 24 by: Igor Urisman Re: Tomcat access log reveals hack attempt: "HEAD /manager/html HTTP/1.0" 404 241112 by: Esmond Pitt 241113 by: Howard W. Smith, Jr. 241114 by: Howard W. Smith, Jr. 241119 by: Jeffrey Janner RE : Tomcat 6.0.35 Crashed again 241115 by: saumil shah Re: Inno Setup Script? 241116 by: James Green 241117 by: Konstantin Kolinko 241118 by: James Green Administrivia: - To post to the list, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: users-digest-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-digest-h...@tomcat.apache.org -- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: RE : Tomcat 6.0.35 Crashed again
you need to do take a look at the loaded JSF webapps and find outwho is acquiring a resource and not closing the resource who is acquiring large amounts of heap and not releasingbe aware any reference to an any object in another class gives the class the right to be placed into PermGenHibernate with cglib proxies are notorious memory hogs awatch your PermGen get pegged when Hibernate and cglib proxies are loadedStatics are another set of culprits of of heap usage Remember all long lived heap objects are eventually placed into Permgen Find the tools to track eden heap, tenured heap and PermGen http://www.integratingstuff.com/2011/07/24/understanding-and-avoiding-the-java-permgen-space-error/ get familiar with taking heap dumps with jmap and analyzing with jhathttp://javarevisited.blogspot.com/2011/05/java-heap-space-memory-size-jvm.html Martin __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. > Date: Thu, 11 Apr 2013 11:40:46 -0400 > Subject: Re: RE : Tomcat 6.0.35 Crashed again > From: smithh032...@gmail.com > To: users@tomcat.apache.org > > On Thu, Apr 11, 2013 at 10:41 AM, Mark H. Wood wrote: > > > Really, no one else can tell you what settings to use. The best we > > can hope for is some accepted rules of thumb *as starting points* for > > further tuning. > > > > > +1 to Dan, Neven, and Mark's responses. Please consider-or-do 'everything' > that they mentioned/recommended. > > I did want to share my java settings for my > currently-considered-a-low-scale JSF web app running on Windows Server 2008 > R2 64bit server with 32GB RAM. > > -XX:HeapDumpPath=D:\apache-tomee-plus-1.6.0-SNAPSHOT\temp > -XX:+HeapDumpOnOutOfMemoryError > -Djava.awt.headless=true > -Dcom.sun.management.jmxremote.port=422 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > -Xms1024m > -Xmx1024m > -XX:MaxPermSize=384m > -XX:+UseTLAB > -XX:+UseConcMarkSweepGC > -XX:+CMSClassUnloadingEnabled > > I am very pleased with the GC performance of my app, and I do like to > monitor the performance of the app via JMX remote connection via Java > Visual VM. My app runs between 200m to 500m, but I am keeping Xms/Xmx=1024m > just to see if I ever get an OOME; so far, so good (never experienced an > OOME), but recently, I did experience some unexpected/unwanted behavior > with one of my @Schedule processes which was attempting to sync some data > from database to/with Google Calendar, and google Calendar service returned > google calendar error 503, and I recognized that the memory got up to 500m, > and the google calendar error 503 did not resolve itself over an hour > (@Schedule executes every 2 to 4 minutes, if error occurs, then data is > appended to the queue for later retry attempt). I never seen that behavior > and I don't know if I will see it again; i wish I would have done a 'heap > dump' instead of a 'stop' tomee/tomcat. Everyday, I listen and read these > questions/responses on tomcat list, and I can't believe that I forgot to do > a 'heap dump'. :( > > Also, please note that I occasionally stop-deploy-and-start tomee/tomcat > almost-on-a-daily-basis to deploy new app-or-configuration-or-library > updates; also, the app or tomee (or tomcat) seem to accumlate threadlocals > over time, and if uptime is > 1 day, then I 'think' I see that memory is > not released, and I think eventually as uptime increases, then the > app/tomee/tomcat will result in OOME. :) > > At any rate, hope this helps. > > Howard
Re: Tomcat 6.0.35 Crashed again
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Dan, On 4/11/13 9:52 AM, Daniel Mikusa wrote: > On Apr 11, 2013, at 1:09 AM, saumil shah wrote: > >> Hello All, We are using Tomcat 6.0.35 for our production system >> with 64 bit JVM (1.6.33) on Windows 2008 R2 SP1. Our physical >> memory is 24gb. Load is ~ 100 concurrent sessions. The Tomcat >> crashed again with OutOfMemoryError: Java heap space error. We >> are using COTS product SAP Business Objects. Upon providing >> Tomcat logs , SAP came with recommendation of increasing the heap >> size . Our current Java params are : -Xms512m -Xmx1024m >> -XX:MaxPermSize=512m SAP's recommendation is to make it ... >> -Xms1024m -Xmx4096m We have ample of free memory on the box ... >> 1a. but the question was heap size as Max 4GB , would it cause >> application delay when java GC is due? It's NOT time-sensitive >> application. > > A bigger heap will generally take longer to clean, but 4GB is not > particularly large. Yes and no: GC time is dictated by the number of live objects (and how much space they take up), not by the size of the heap space or the amount of garbage. Moving a chunk of data from one heap space to another is fairly quick, but if there are millions of tiny moves (8 bytes here, 8 bytes there), that's what takes the time. If you have a 10MiB byte array, it's going to move faster than 1M 8-byte pointers randomly-scattered throughout the heap space. > If you're concerned about the performance of the garbage collector, > I would suggest that you enable garbage collection logging. This > adds very little overhead, simply logging an entry for every > garbage collection that occurs. Each entry will indicate memory > before and after the GC, plus how much time was spent performing > the collection. You can use that to analyze the performance for > your application. Again, yes and no: when focusing on performance of the garbage collector, enabling GC logging is pretty much the first step you want to take. On the other hand, performance doesn't seem to be an issue, here: (lack of) stability is the current issue. >> 1b. Also, when putting MAx as 4GB , does it mean JVM will make >> base effort to get 4GB MAX memory for heap , or is that >> guaranteed? Based on my reading, it appeared that only Min, was >> guaranteed. > > The JVM will start at -Xms and, if needed, expand up to -Xmx. > >> 1c. Is it then recommended to have Min and Max as 4GB... so that >> Heap doesn't have to resize and 4GB is guaranteed at the time >> of initialization? > > It depends. This can be good because it will prevent a few full > GC's as the JVM expands from -Xms to -Xmx. However, it can also > waste memory if you allocate too much. +1 Re-sizing the heap does take time. If you expect to fully-utilize a 4GiB heap, you may as well allocate it all at once to avoid heap-thrashing just to get up to 4GiB. - -Xms4096M -Xmx4096M On the other hand, the OP's problem is likely to be an actual *leak*, in which case a huge heap only prolongs the inevitable. If I were debugging this on a test rig, the first thing I would do is *reduce* the size of the heap to make heap analysis easier. >> 3. Once JConsole and/or VisualVM is setup ... I can look at >> Heap/Permgen , memory , threads etc.. I was wondering how do >> I see memory leaks and what is causing it , using these tools? > > Spotting memory leaks is tricky and I'm not sure that what you're > proposing will be much help. You could certainly look at the live > graphs and watch for a trend in memory usage increasing, but if > the leak is small it would be very hard to spot. > > I would suggest that you add -XX:+HeapDumpOnOutOfMemoryError to > your JVM options. This will create a heap dump when you encounter > an OOME. You can then view that heap dump in jvisualvm or a > profiler to see what was consuming the memory. +1 Heap dumps are no fun to analyze, but you might notice that you have a whole bunch of objects you didn't expect to be sitting-around in the heap (like java.sql.Connection, or huge byte arrays, or whatever). I use YourKit (full disclosure: I get a free license as an ASF committer) and it's got nice utilities for scanning heap snapshots for some obvious potential problems (heap hogs, duplicate objects, arrays with nothing but nulls in them, etc.). Most heap analyzers have tools to let you find out how many objects of a particular type you have, etc. and that may give you a quick idea of what may be going on. If this is the first time you've ever looked at a heap dump, don't be scared when you see the number of byte[], char[], and String objects are still reachable ;) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRZy8cAAoJEBzwKT+lPKRYV8wQAJ9fNdj7ARfzro6y3QHd7w/Y NcnnynPMQbgJK52RBWFDP4TrqxKMeeF+NpTm14i4yS2x+S6qQvjZp3NgMjW+0h7H nFUHWgYWumN7tHUcS213/
Re: Tomcat access log reveals hack attempt: "HEAD /manager/html HTTP/1.0" 404
2013/4/12 Christopher Schultz : > >> The attacker installed a viral servlet application that killed the >> server completely, we had to rebuild it. > > I -- like most people I would guess -- don't run under a > SecurityManager, but doing so can significantly limit the damage that > a rogue webapp can do. > If you do not trust your applications then it is recommended to run with . http://tomcat.apache.org/tomcat-7.0-doc/security-howto.html#Host I think there are not enough checks in place to avoid abuse if webapp is able to provide its own context.xml file, even if you run with a SecurityManager. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat access log reveals hack attempt: "HEAD /manager/html HTTP/1.0" 404
On Thu, Apr 11, 2013 at 4:39 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Jeffrey, > > On 4/11/13 9:47 AM, Jeffrey Janner wrote: > >> -Original Message- From: Howard W. Smith, Jr. > >> [mailto:smithh032...@gmail.com] Sent: Wednesday, April 10, 2013 > >> 7:35 PM To: Esmond Pitt Cc: Tomcat Users List Subject: Re: Tomcat > >> access log reveals hack attempt: "HEAD /manager/html HTTP/1.0" > >> 404 > >> > >> On Wed, Apr 10, 2013 at 8:21 PM, Esmond Pitt > >> wrote: > >> > >>> We had lots of these and finally an attack last year on a > >>> Tomcat > >> where > >>> the manager password somehow hadn't been changed. The attacker > >>> installed a viral servlet application that killed the server > >>> completely, we had to rebuild it. > >>> > >>> We: > >>> > >>> - Hid the Tomcat behind an Apache HTTPD on port 80. - Closed > >>> port 8080, indeed removed all the HTTP Connectors from > >> Tomcat > >>> and just used AJP connectors running on 127.0.0.1/2/3/4/..., > >>> all on the same port for simplicity, so there is no zero direct > >>> access to Tomcat from the outside - Configured Apache HTTPD for > >>> LDAP authentication via an OpenLDAP server that in turn is > >>> configured via the Password Policy overlay for finite (5 I > >>> think) password retries before locking out the account - > >>> required a very restricted LDAP group membership for access to > >>> /manager (and the other Tomcat builtins). > >>> > >>> No recurrence, not even an attempt. I think actually closing > >>> port > >> 8080 > >>> may have played the biggest part in all this. > >>> > >>> EJP > >>> > >>> > >> +1 I like what you all did! I'm currently not using Apache > >> HTTPD, 'yet'. > >> > >> Before I start TomEE/tomcat, I always copy my edited version of > >> tomee/tomcat's user file, and I have a strong password in place. > >> when I first started using TomEE, and when I had port 3389 open > >> on my Windows Server 2008 'development server', I saw someone > >> connect to the tomee and tomcat manager apps, and they tried > >> 'many' times to login to those manager app pages. > >> > >> I LOL at them, because even though the manager apps were > >> available, i already beat them to the punch, because I secured > >> tomee/tomcat by commenting out users and/or user groups in the > >> user file, and created my own custom user that had a strong > >> password. So, after I saw those blatent-and-sorry-hacker > >> attempts, I resolved that by removing manager apps whenever I > >> install new version of tomee/tomcat. Problem solved!!! :) And > >> yes, i eventually, closed port 3389 on my router, since I really > >> don't need it since I am in the office 99.9% of the time > >> doing my work. Sometimes, if I have to travel somewhere or sit in > >> waiting room, while my vehicle is being service, I do get tempted > >> to open 3389 port on my router and do some work at that time. :) > >> > > > > FYI, Howard, this is why they invented VPN technology. > > +1 > > OpenVPN is cheap and relatively easy to set up. > Interesting. Thanks Chris. > > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJRZx+eAAoJEBzwKT+lPKRYTD8P/RYPp4fq476XkWWnBQ+Z5hQn > sGNkos89wTDvMWDTSaDclZ3zcc8RDGDBq4Mv/iN6TXev9ztZAiw5iQIbWqg1TiMx > sEgaL++mtvC825epomP8vzxrc7EmAlM/iTLsnUxIxJSFXp93/ntLWy4drPPERxNr > nXoRBNL9pdwAMln4e693I2TUsezH3zr+bppjfe3pzKWk0JU/Y1+Cp/XycwPKklwK > qNhtgztqrL7URx28r/GPQ6/yUEoXzEe4PFBB+rZ7XyDqPlH30XmnUBXAU+B0Lr1D > wekhHVSjVzl4UhgiAFxm1VF4FAuAG/Lvuia7Z4Jt074H7UaGVfsyauurWFn5JC0l > 8NDVlBqRufHHmUPgZSIctR8vyqp4vbRKCcdL5CdXQ9TgScEWI+cVYzi4VjVz4kyR > FRKhMZXC4K8lqvMkecLNjNLISp8KhAaGkM9sffzOLzWyqxPG8u7us26MScBKoAaJ > 60gTJcDZ5jU0mywhJrGBK+X9ceKEIX0fafSiPbQ64Rb/MNxgkD9r92AiE4Ycslbg > cAEHxioCrrTumCVeFCb9b9a+ZMXVw0LlBtUUeo8V5q/9KXTfQ5WFhXKPadN6tbP3 > ERGTFXZUU+8Kbe5ziv5m/039RUaOXnAFLUN46JcNfT2sKn/KkirV9DifxmnP3roh > E/MwnaE4+YWdG5WSdvRa > =28Nh > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat access log reveals hack attempt: "HEAD /manager/html HTTP/1.0" 404
On Thu, Apr 11, 2013 at 9:47 AM, Jeffrey Janner wrote: > > -Original Message- > > From: Howard W. Smith, Jr. [mailto:smithh032...@gmail.com] > > Sent: Wednesday, April 10, 2013 7:35 PM > > To: Esmond Pitt > > Cc: Tomcat Users List > > Subject: Re: Tomcat access log reveals hack attempt: "HEAD > > /manager/html HTTP/1.0" 404 > > > > On Wed, Apr 10, 2013 at 8:21 PM, Esmond Pitt > > wrote: > > > > > We had lots of these and finally an attack last year on a Tomcat > > where > > > the manager password somehow hadn't been changed. The attacker > > > installed a viral servlet application that killed the server > > > completely, we had to rebuild it. > > > > > > We: > > > > > > - Hid the Tomcat behind an Apache HTTPD on port 80. > > > - Closed port 8080, indeed removed all the HTTP Connectors from > > Tomcat > > > and just used AJP connectors running on 127.0.0.1/2/3/4/..., all on > > > the same port for simplicity, so there is no zero direct access to > > > Tomcat from the outside > > > - Configured Apache HTTPD for LDAP authentication via an OpenLDAP > > > server that in turn is configured via the Password Policy overlay for > > > finite (5 I > > > think) password retries before locking out the account > > > - required a very restricted LDAP group membership for access to > > > /manager (and the other Tomcat builtins). > > > > > > No recurrence, not even an attempt. I think actually closing port > > 8080 > > > may have played the biggest part in all this. > > > > > > EJP > > > > > > > > +1 I like what you all did! I'm currently not using Apache HTTPD, > > 'yet'. > > > > Before I start TomEE/tomcat, I always copy my edited version of > > tomee/tomcat's user file, and I have a strong password in place. when I > > first started using TomEE, and when I had port 3389 open on my Windows > > Server 2008 'development server', I saw someone connect to the tomee > > and tomcat manager apps, and they tried 'many' times to login to those > > manager app pages. > > > > I LOL at them, because even though the manager apps were available, i > > already beat them to the punch, because I secured tomee/tomcat by > > commenting out users and/or user groups in the user file, and created > > my own custom user that had a strong password. So, after I saw those > > blatent-and-sorry-hacker attempts, I resolved that by removing manager > > apps whenever I install new version of tomee/tomcat. Problem solved!!! > > :) And yes, i eventually, closed port 3389 on my router, since I > > really don't need it since I am in the office 99.9% of the time > > doing my work. > > Sometimes, if I have to travel somewhere or sit in waiting room, while > > my vehicle is being service, I do get tempted to open 3389 port on my > > router and do some work at that time. :) > > > > FYI, Howard, this is why they invented VPN technology. > > True/agreed! Tried it years ago (when, evidently, our network was slower), and we were completely turned off by the really really really slow connection. Honestly, we've been managing well, for years, without VPN. Thanks Jeffrey! > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat access log reveals hack attempt: "HEAD /manager/html HTTP/1.0" 404
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jeffrey, On 4/11/13 9:47 AM, Jeffrey Janner wrote: >> -Original Message- From: Howard W. Smith, Jr. >> [mailto:smithh032...@gmail.com] Sent: Wednesday, April 10, 2013 >> 7:35 PM To: Esmond Pitt Cc: Tomcat Users List Subject: Re: Tomcat >> access log reveals hack attempt: "HEAD /manager/html HTTP/1.0" >> 404 >> >> On Wed, Apr 10, 2013 at 8:21 PM, Esmond Pitt >> wrote: >> >>> We had lots of these and finally an attack last year on a >>> Tomcat >> where >>> the manager password somehow hadn't been changed. The attacker >>> installed a viral servlet application that killed the server >>> completely, we had to rebuild it. >>> >>> We: >>> >>> - Hid the Tomcat behind an Apache HTTPD on port 80. - Closed >>> port 8080, indeed removed all the HTTP Connectors from >> Tomcat >>> and just used AJP connectors running on 127.0.0.1/2/3/4/..., >>> all on the same port for simplicity, so there is no zero direct >>> access to Tomcat from the outside - Configured Apache HTTPD for >>> LDAP authentication via an OpenLDAP server that in turn is >>> configured via the Password Policy overlay for finite (5 I >>> think) password retries before locking out the account - >>> required a very restricted LDAP group membership for access to >>> /manager (and the other Tomcat builtins). >>> >>> No recurrence, not even an attempt. I think actually closing >>> port >> 8080 >>> may have played the biggest part in all this. >>> >>> EJP >>> >>> >> +1 I like what you all did! I'm currently not using Apache >> HTTPD, 'yet'. >> >> Before I start TomEE/tomcat, I always copy my edited version of >> tomee/tomcat's user file, and I have a strong password in place. >> when I first started using TomEE, and when I had port 3389 open >> on my Windows Server 2008 'development server', I saw someone >> connect to the tomee and tomcat manager apps, and they tried >> 'many' times to login to those manager app pages. >> >> I LOL at them, because even though the manager apps were >> available, i already beat them to the punch, because I secured >> tomee/tomcat by commenting out users and/or user groups in the >> user file, and created my own custom user that had a strong >> password. So, after I saw those blatent-and-sorry-hacker >> attempts, I resolved that by removing manager apps whenever I >> install new version of tomee/tomcat. Problem solved!!! :) And >> yes, i eventually, closed port 3389 on my router, since I really >> don't need it since I am in the office 99.9% of the time >> doing my work. Sometimes, if I have to travel somewhere or sit in >> waiting room, while my vehicle is being service, I do get tempted >> to open 3389 port on my router and do some work at that time. :) >> > > FYI, Howard, this is why they invented VPN technology. +1 OpenVPN is cheap and relatively easy to set up. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRZx+eAAoJEBzwKT+lPKRYTD8P/RYPp4fq476XkWWnBQ+Z5hQn sGNkos89wTDvMWDTSaDclZ3zcc8RDGDBq4Mv/iN6TXev9ztZAiw5iQIbWqg1TiMx sEgaL++mtvC825epomP8vzxrc7EmAlM/iTLsnUxIxJSFXp93/ntLWy4drPPERxNr nXoRBNL9pdwAMln4e693I2TUsezH3zr+bppjfe3pzKWk0JU/Y1+Cp/XycwPKklwK qNhtgztqrL7URx28r/GPQ6/yUEoXzEe4PFBB+rZ7XyDqPlH30XmnUBXAU+B0Lr1D wekhHVSjVzl4UhgiAFxm1VF4FAuAG/Lvuia7Z4Jt074H7UaGVfsyauurWFn5JC0l 8NDVlBqRufHHmUPgZSIctR8vyqp4vbRKCcdL5CdXQ9TgScEWI+cVYzi4VjVz4kyR FRKhMZXC4K8lqvMkecLNjNLISp8KhAaGkM9sffzOLzWyqxPG8u7us26MScBKoAaJ 60gTJcDZ5jU0mywhJrGBK+X9ceKEIX0fafSiPbQ64Rb/MNxgkD9r92AiE4Ycslbg cAEHxioCrrTumCVeFCb9b9a+ZMXVw0LlBtUUeo8V5q/9KXTfQ5WFhXKPadN6tbP3 ERGTFXZUU+8Kbe5ziv5m/039RUaOXnAFLUN46JcNfT2sKn/KkirV9DifxmnP3roh E/MwnaE4+YWdG5WSdvRa =28Nh -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat access log reveals hack attempt: "HEAD /manager/html HTTP/1.0" 404
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Esmond, On 4/10/13 8:21 PM, Esmond Pitt wrote: > We had lots of these and finally an attack last year on a Tomcat > where the manager password somehow hadn't been changed. Note that the manager webapp has no default passwords, so I wonder what you mean when you say it "hadn't been changed". There are examples in conf/tomcat-users.xml but they are all commented-out. You would have had to intentionally enable the "default" password. > The attacker installed a viral servlet application that killed the > server completely, we had to rebuild it. I -- like most people I would guess -- don't run under a SecurityManager, but doing so can significantly limit the damage that a rogue webapp can do. > We: > > - Hid the Tomcat behind an Apache HTTPD on port 80. Did you also remove manager webapp access through httpd? Otherwise, this doesn't actually do anything to help. > - Closed port 8080, indeed removed all the HTTP Connectors from > Tomcat and just used AJP connectors running on 127.0.0.1/2/3/4/..., > all on the same port for simplicity, so there is no zero direct > access to Tomcat from the outside +1, though I would run Apache httpd and Tomcat on different hosts, so localhost-binding is not possible unless you are doing something like stunnel (which also might be a good idea if you are traversing an untrusted network). > - Configured Apache HTTPD for LDAP authentication via an OpenLDAP > server that in turn is configured via the Password Policy overlay > for finite (5 I think) password retries before locking out the > account +2 -- both good ideas: central access control (LDAP) and enabling lockout mechanism. Note that Tomcat's lockout mechanism is fairly primitive and easy to game. > - required a very restricted LDAP group membership for access to > /manager (and the other Tomcat builtins). +1 hooray for role-based permissions! > No recurrence, not even an attempt. I think actually closing port > 8080 may have played the biggest part in all this. Would you be willing to review the Tomcat documentation on "securing Tomcat" and make a few comments? It could always use some additional tips: http://tomcat.apache.org/tomcat-7.0-doc/security-howto.html http://wiki.apache.org/tomcat/FAQ/Security You can sign-up for the wiki yourself and make any changes you want. If you want to modify the "official" documentation, create a Bugzilla enhancement request and (please!) include a patch. I'm sure it will go right in. Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRZx7AAAoJEBzwKT+lPKRY+3MP/3c8kZZU43cMaxkXTi/ELXha Hv6rAeGz4nnpMNn2C002cTRgZ39vomUXYdsLMnNIshn05JDVIZmLLoUXk6UzY9go uH0QdAubBxhvwC/CWeLjUuSjy/Ei4vKeB7xJNw/FQ2xXEt47FWv36e0vgxOyluX+ gbkB3KQlN6PXtQENGvkOGT5oWLK9M7WUydGSWq9lXR+akwWeL3jWRAlLl6bHYybQ n70c5wq/rJbEj+k9yCHsMZvPabYs5ejsz6wHvvw4Emrxcp4LVVjCuY2Z87Yhdtb4 B43tF48be9GUZCXDvtIjiS5phHMhpqyJakHuZ7GSvzDIeuiNZ96XuoDkIG1bwWjf Z5SMCSjkSPqDKJ1cXcd8AaSYgI2C3KQnuFrbTD7bVqQHOeq7RJZp3+xE0IUNPl+V H2PNpUfXD9BDbPiiDt8bcgvcrImejW0RDumQ2fwbTVvt4OaybVsMUsVFW8lUtw3A YhvFn/WCEdR8VaY9PkqYm84BVMsQJBbBdb5clYiAtVQRky1NPS+hcIihnf85DkNU vr6rv/oK0aMXAamwUagmRe5OjTHuHczERPYgEUMpppnlXuNV1mLxBib8+HInGg3/ Y5i28tTd7fS5uo7/CZv+9uEZdDUO7utWGT0W+gBaIkh35/yZI5a1l5wi0szYduQe t3nftQXUTCYtK1QNwKND =3s6Q -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: RE : Tomcat 6.0.35 Crashed again
On Thu, Apr 11, 2013 at 10:41 AM, Mark H. Wood wrote: > Really, no one else can tell you what settings to use. The best we > can hope for is some accepted rules of thumb *as starting points* for > further tuning. > > +1 to Dan, Neven, and Mark's responses. Please consider-or-do 'everything' that they mentioned/recommended. I did want to share my java settings for my currently-considered-a-low-scale JSF web app running on Windows Server 2008 R2 64bit server with 32GB RAM. -XX:HeapDumpPath=D:\apache-tomee-plus-1.6.0-SNAPSHOT\temp -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=422 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xms1024m -Xmx1024m -XX:MaxPermSize=384m -XX:+UseTLAB -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled I am very pleased with the GC performance of my app, and I do like to monitor the performance of the app via JMX remote connection via Java Visual VM. My app runs between 200m to 500m, but I am keeping Xms/Xmx=1024m just to see if I ever get an OOME; so far, so good (never experienced an OOME), but recently, I did experience some unexpected/unwanted behavior with one of my @Schedule processes which was attempting to sync some data from database to/with Google Calendar, and google Calendar service returned google calendar error 503, and I recognized that the memory got up to 500m, and the google calendar error 503 did not resolve itself over an hour (@Schedule executes every 2 to 4 minutes, if error occurs, then data is appended to the queue for later retry attempt). I never seen that behavior and I don't know if I will see it again; i wish I would have done a 'heap dump' instead of a 'stop' tomee/tomcat. Everyday, I listen and read these questions/responses on tomcat list, and I can't believe that I forgot to do a 'heap dump'. :( Also, please note that I occasionally stop-deploy-and-start tomee/tomcat almost-on-a-daily-basis to deploy new app-or-configuration-or-library updates; also, the app or tomee (or tomcat) seem to accumlate threadlocals over time, and if uptime is > 1 day, then I 'think' I see that memory is not released, and I think eventually as uptime increases, then the app/tomee/tomcat will result in OOME. :) At any rate, hope this helps. Howard
Re: RE : Tomcat 6.0.35 Crashed again
Really, no one else can tell you what settings to use. The best we can hope for is some accepted rules of thumb *as starting points* for further tuning. I'd suggest choosing a tool that lets you easily monitor the memory pools, and checking it frequently as you adjust the pool sizes. If your applications are not leaking memory, the pools should each expand to a certain size and then tend to stay there. I would set each pool's initial size slightly larger than its steady-state size, and set some additional headroom on its maximum size to cope with unpredictable demand bursts. (Actually I would leave most of them alone and just tune the ones that seem significantly out-of-tune.) I suggest continuing to monitor memory behavior on a regular basis. Your load probably varies over time, and different versions of code behave somewhat differently. I have a repeating reminder on my calendar to check my Tomcat instances weekly. I use PsiProbe for peeking inside Tomcat, but there are a number of other good tools. As your Tomcat tuning progresses, you'll find what its overall size "ought to be", and can then consider tuning and perhaps resizing the surrounding system. A well-tuned servlet container running in a poorly-tuned OS or undersized hardware will still underperform. The general plan here is the same: start with an educated guess, observe, adjust, monitor -- Mark H. Wood, Lead System Programmer mw...@iupui.edu There's an app for that: your browser pgpH1Gyl2LNzZ.pgp Description: PGP signature
Re: RE : Tomcat 6.0.35 Crashed again
Saumil, On Thu, Apr 11, 2013 at 1:09 AM, saumil shah wrote: > Hello All, > We are using Tomcat 6.0.35 for our production system with 64 bit JVM > (1.6.33) on Windows 2008 R2 SP1. Our physical memory is 24gb. Load is ~ 100 > concurrent sessions. > The Tomcat crashed again with OutOfMemoryError: Java heap space error. We > are using COTS product SAP Business Objects. Upon providing Tomcat logs , > SAP came with recommendation of increasing the heap size . Our current > Java params are : > -Xms512m -Xmx1024m > -XX:MaxPermSize=512m > SAP's recommendation is to make it ... > -Xms1024m -Xmx4096m We have ample of free memory on the box ... > 1a. but the question was heap size as Max 4GB , would it cause application > delay when java GC is due? It's NOT time-sensitive application.1b. Also, > when putting MAx as 4GB , does it mean JVM will make base effort to get 4GB > MAX memory for heap , or is that guaranteed? Based on my reading, it > appeared that only Min, was guaranteed.1c. Is it then recommended to have > Min and Max as 4GB ... so that Heap doesn't have to resize and 4GB is > guaranteed at the time of initialization?1d. I have Tomcat service > installer. I have provided Min and Max heap size values under Tomcat > --> Java tab --> Initial Memory Pool and Maximum memory pool .. I am > assuming that means Min and Max values for Heap.is this correct? > You are correct. When starting JVM you are providing JVM parameters as part of the command line. Usually, we define that in some script (Tomcat's best way is to place these in setenv.bat). -Xms represents initial heap size -Xmx represents the maximum size heap can grow to Now, you might see JVM process takes more memory than Xmx value, because JVM needs to allocate memory not only for heap, but some other things as well, e.g. stack for each thread (-Xss value). If the stack is too small you would get StackOverflowError, if you have lot of threads (and stack is fairly large) you will run out of memory. Now, I don't think that's your case. You have 24G on the machine. Read more about JVM tuning here: http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html "...Large server applications often experience two problems with these defaults. One is slow startup, because the initial heap is small and must be resized over many major collections. A more pressing problem is that the default maximum heap size is unreasonably small for most server applications. The rules of thumb for server applications are: - Unless you have problems with pauses, try granting as much memory as possible to the virtual machine. The default size (64MB) is often too small. - Setting -Xms and -Xmx to the same value increases predictability by removing the most important sizing decision from the virtual machine. However, the virtual machine is then unable to compensate if you make a poor choice. - In general, increase the memory as you increase the number of processors, since allocation can be parallelized." Of course, it really comes down the application memory usage. How is your application written? How does it behave under heavy load? How are your objects being created, how many, how often, how big? How do you setup GC, what heuristics/strategies did you setup, etc... Often times, you would see people set -Xmx and -Xms to the same value (e.g. 4G) this is to reduce the resizing of the initial heap multiple times as your heap consumption grows. However, setting the heap size too big, your GC has much more memory to manage, which makes the GC overhead impact the performance of the application... There is no cookie-cutter solution that fits all. JVM optimization is trial-and-error process. One is sure - you have to have a good testing environment where you can reasonably reproduce loads your application usually gets (heavy load, light load, etc...) (Where are you setting these values Xmx and Xms btw?) 2. When trying to monitor Tomcat , using JConsole and VisualVM . do I > need to add JMX agent and port in Java tab of Tomcat and then restart > , or is it possible to Monitor tomcat , without rebooting it ? Since, it > would not be possible to re-boot tomcat in production for it. > Short answer is no. If you can login into the machine where Tomcat runs, you can run jconsole/jvisualvm locally and there is no need to open JMX management ports. However, if you want to connect to remote JVM processes (ie. JVM running on remote machines, e.g. production JVMs) you need to open up the JMX management ports using appropriate switches (again, these are set setenv.bat) More details here: http://docs.oracle.com/javase/tutorial/jmx/remote/jconsole.html 3. Once JConsole and/or VisualVM is setup ... I can look at Heap/Permgen , > memory , threads etc.. I was wondering how do I see memory leaks and > what is causing it , using these tools? > Memory leak is situation when the objects keep getting stuck in memory unnecessary. You can't really detect these things
Re: Tomcat 6.0.35 Crashed again
On Apr 11, 2013, at 1:09 AM, saumil shah wrote: > Hello All, > We are using Tomcat 6.0.35 for our production system with 64 bit JVM (1.6.33) > on Windows 2008 R2 SP1. Our physical memory is 24gb. Load is ~ 100 concurrent > sessions. > The Tomcat crashed again with OutOfMemoryError: Java heap space error. We are > using COTS product SAP Business Objects. Upon providing Tomcat logs , > SAP came with recommendation of increasing the heap size . Our current Java > params are : > -Xms512m -Xmx1024m > -XX:MaxPermSize=512m > SAP's recommendation is to make it ... > -Xms1024m -Xmx4096m > We have ample of free memory on the box ... > 1a. but the question was heap size as Max 4GB , would it cause application > delay when java GC is due? It's NOT time-sensitive application. A bigger heap will generally take longer to clean, but 4GB is not particularly large. If you're concerned about the performance of the garbage collector, I would suggest that you enable garbage collection logging. This adds very little overhead, simply logging an entry for every garbage collection that occurs. Each entry will indicate memory before and after the GC, plus how much time was spent performing the collection. You can use that to analyze the performance for your application. > 1b. Also, when putting MAx as 4GB , does it mean JVM will make base effort to > get 4GB MAX memory for heap , or is that guaranteed? Based on my reading, it > appeared that only Min, was guaranteed. The JVM will start at -Xms and, if needed, expand up to -Xmx. > 1c. Is it then recommended to have Min and Max as 4GB... so that Heap doesn't > have to resize and 4GB is guaranteed at the time of initialization? It depends. This can be good because it will prevent a few full GC's as the JVM expands from -Xms to -Xmx. However, it can also waste memory if you allocate too much. > 1d. I have Tomcat service installer. I have provided Min and Max heap > size values under Tomcat --> Java tab --> Initial Memory Pool and Maximum > memory pool .. I am assuming that means Min and Max values for > Heap.is this correct? Yes, I believe so. > 2. When trying to monitor Tomcat , using JConsole and VisualVM . do I > need to add JMX agent and port in Java tab of Tomcat and then restart , > or is it possible to Monitor tomcat , without rebooting it ? Since, it would > not be possible to re-boot tomcat in production for it. Normally when you run jconsole or jvisualvm on the same machine as your Tomcat instance, you can connect without needing any additional configuration. This is a special case because the two processes are running on the same machine. However, I seem to remember needing to configure JMX when running Tomcat as a Windows Service. Perhaps one of the Windows guys on this list can clarify. > 3. Once JConsole and/or VisualVM is setup ... I can look at Heap/Permgen , > memory , threads etc.. I was wondering how do I see memory leaks and what > is causing it , using these tools? Spotting memory leaks is tricky and I'm not sure that what you're proposing will be much help. You could certainly look at the live graphs and watch for a trend in memory usage increasing, but if the leak is small it would be very hard to spot. I would suggest that you add -XX:+HeapDumpOnOutOfMemoryError to your JVM options. This will create a heap dump when you encounter an OOME. You can then view that heap dump in jvisualvm or a profiler to see what was consuming the memory. > 4. Also, I am seeing that Tomcat is deploying the webapps on start-up is > there anyway to avoid Tomcat to deploy the application on start-upas the > applications are already deployed and exploded? > Example is below :Apr 5, 2013 7:17:27 PM > org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web > application archive AnalyticalReporting.war Tomcat has to deploy your application when it starts or your user's won't be able to access it. This is normal and expected behavior. > Apr 5, 2013 7:17:28 PM org.apache.catalina.core.StandardContext > addApplicationListenerINFO: The listener > "com.sun.faces.config.ConfigureListener" is already configured for this > context. The duplicate definition has been ignored. Looks like you might have a duplicate definition in your application's web.xml. You should remove it, but you don't have to. This is just Tomcat telling you that the duplicate was ignored. Dan > > As you can tell , I am novice to Tomcat. Please let me know if I missed on > something. > Appreciate all your help. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat access log reveals hack attempt: "HEAD /manager/html HTTP/1.0" 404
> -Original Message- > From: Howard W. Smith, Jr. [mailto:smithh032...@gmail.com] > Sent: Wednesday, April 10, 2013 7:35 PM > To: Esmond Pitt > Cc: Tomcat Users List > Subject: Re: Tomcat access log reveals hack attempt: "HEAD > /manager/html HTTP/1.0" 404 > > On Wed, Apr 10, 2013 at 8:21 PM, Esmond Pitt > wrote: > > > We had lots of these and finally an attack last year on a Tomcat > where > > the manager password somehow hadn't been changed. The attacker > > installed a viral servlet application that killed the server > > completely, we had to rebuild it. > > > > We: > > > > - Hid the Tomcat behind an Apache HTTPD on port 80. > > - Closed port 8080, indeed removed all the HTTP Connectors from > Tomcat > > and just used AJP connectors running on 127.0.0.1/2/3/4/..., all on > > the same port for simplicity, so there is no zero direct access to > > Tomcat from the outside > > - Configured Apache HTTPD for LDAP authentication via an OpenLDAP > > server that in turn is configured via the Password Policy overlay for > > finite (5 I > > think) password retries before locking out the account > > - required a very restricted LDAP group membership for access to > > /manager (and the other Tomcat builtins). > > > > No recurrence, not even an attempt. I think actually closing port > 8080 > > may have played the biggest part in all this. > > > > EJP > > > > > +1 I like what you all did! I'm currently not using Apache HTTPD, > 'yet'. > > Before I start TomEE/tomcat, I always copy my edited version of > tomee/tomcat's user file, and I have a strong password in place. when I > first started using TomEE, and when I had port 3389 open on my Windows > Server 2008 'development server', I saw someone connect to the tomee > and tomcat manager apps, and they tried 'many' times to login to those > manager app pages. > > I LOL at them, because even though the manager apps were available, i > already beat them to the punch, because I secured tomee/tomcat by > commenting out users and/or user groups in the user file, and created > my own custom user that had a strong password. So, after I saw those > blatent-and-sorry-hacker attempts, I resolved that by removing manager > apps whenever I install new version of tomee/tomcat. Problem solved!!! > :) And yes, i eventually, closed port 3389 on my router, since I > really don't need it since I am in the office 99.9% of the time > doing my work. > Sometimes, if I have to travel somewhere or sit in waiting room, while > my vehicle is being service, I do get tempted to open 3389 port on my > router and do some work at that time. :) > FYI, Howard, this is why they invented VPN technology. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Inno Setup Script?
A follow-up with the solution. This works: C:\Program Files\Our Local Services\tomcat\bin\tomcat7.exe //IS//Tomcat7 --DisplayName="Apache Tomcat 7" --JavaHome="C:\Program Files\Our Local Services\tomcat\..\java-7-32bit" --Install="C:\Program Files\Our Local Services\tomcat\bin\tomcat7.exe" --Jvm="C:\Program Files\Our Local Services\tomcat\..\java-7-32bit\bin\client\jvm.dll" --JvmOptions="-Djava.io.tmpdir=C:\Program Files\Our Local Services\tomcat\temp;-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager;-Djava.util.logging.config.file=C:\Program Files\Our Local Services\tomcat\conf\logging.properties;" --Classpath="C:\Program Files\Our Local Services\tomcat\bin\bootstrap.jar;C:\Program Files\Our Local Services\tomcat\bin\tomcat-juli.jar" --StartMode=jvm --StopMode=jvm --StartClass=org.apache.catalina.startup.Bootstrap --StartParams=start --StopClass=org.apache.catalina.startup.Bootstrap --StopParams=stop --LogPath="C:\Program Files\Our Local Services\tomcat\logs" --LogLevel=DEBUG This does not: C:\Program Files\Our Local Services\tomcat\bin\tomcat7.exe //IS//Tomcat7 --DisplayName="Apache Tomcat 7" --JavaHome="C:\Program Files\Our Local Services\tomcat\..\java-7-32bit\" --Install="C:\Program Files\Our Local Services\tomcat\bin\tomcat7.exe" --Jvm="C:\Program Files\Our Local Services\tomcat\..\java-7-32bit\bin\client\jvm.dll" --JvmOptions="-Djava.io.tmpdir=C:\Program Files\Our Local Services\tomcat\temp;-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager;-Djava.util.logging.config.file=C:\Program Files\Our Local Services\tomcat\conf\logging.properties;" --Classpath="C:\Program Files\Our Local Services\tomcat\bin\bootstrap.jar;C:\Program Files\Our Local Services\tomcat\bin\tomcat-juli.jar" --StartMode=jvm --StopMode=jvm --StartClass=org.apache.catalina.startup.Bootstrap --StartParams=start --StopClass=org.apache.catalina.startup.Bootstrap --StopParams=stop --LogPath="C:\Program Files\Our Local Services\tomcat\logs" --LogLevel=DEBUG The difference is that in the working line the value for JavaHome does NOT end with a backslash. This was finally identified by comparing registry data between our custom installer and the official one. Now to build much the same but for HSQL... James On 11 April 2013 10:08, James Green wrote: > Getting quite frustrated now. > > Here's the line the installer is now running: > > C:\Program Files\Our Local Services\tomcat\bin\tomcat7.exe //IS//Tomcat7 > --DisplayName="Apache Tomcat 7" --JavaHome="C:\Program Files\Our Local > Services\tomcat\..\java-7-32bit\" --Install="C:\Program Files\Our Local > Services\tomcat\bin\tomcat7.exe" --Jvm="C:\Program Files\Our Local > Services\tomcat\..\java-7-32bit\bin\client\jvm.dll" > --JvmOptions="-Djava.io.tmpdir=C:\Program Files\Our Local > Services\tomcat\temp;-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager;-Djava.util.logging.config.file=C:\Program > Files\Our Local Services\tomcat\conf\logging.properties;" > --Classpath="C:\Program Files\Our Local > Services\tomcat\bin\bootstrap.jar;C:\Program Files\Our Local > Services\tomcat\bin\tomcat-juli.jar" --StartMode=jvm --StopMode=jvm > --StartClass=org.apache.catalina.startup.Bootstrap --StartParams=start > --StopClass=org.apache.catalina.startup.Bootstrap --StopParams=stop > --LogPath="C:\Program Files\Our Local Services\tomcat\logs" --LogLevel=DEBUG > > If I subsequently run tomcat7w.exe, the JVM home is not set, nor are lots > of other matters. Again, we're seeing error code 1 from Tomcat. > > EIther I'm doing something dumb or this just doesn't work as described. > > James >
Re: Inno Setup Script?
2013/4/10 James Green : > On 10 April 2013 12:47, Konstantin Kolinko wrote: >>> >> C:\Program Files\Our Local Services\tomcat\bin\tomcat7. >> exe //IS//Tomcat7 >> --DisplayName="Apache Tomcat 7" --JavaHome="C:\Program Files\Our Local >> Services\tomcat\..\java-7-32bit\" --Install="C:\Program Files\Our Local >> Services\tomcat\bin\tomcat7.exe" --Jvm="C:\Program Files\Our Local >> Services\tomcat\..\java-7-32bit\bin\client\jvm.dll" --StartMode=jvm >> --StopMode=jvm --StartClass=org.apache.catalina.startup.Bootstrap >> --StartParams=start --StopClass=org.apache.catalina.startup.Bootstrap >> --StopParams=stop >> >> That command is not enough. There is no classpath, there is no logging >> configuration (neither for the service itself nor for >> java.util.logging). There is no configuration for Tomcat itself >> (catalina.home, catalina.base). >> > > Hmm I was merely following http://tomcat.apache.org/tomcat-7.0-doc/windows-service-howto.html#Installing_services > to > the letter. This does not specify a classpath unless you look under the > section "Updating services". The other matters are also not mentioned. > > Where might I find the defaults I need? > 1. The first sentence in that section: "The safest way to manually install the service is to use the provided service.bat script." That is the recommended way if you are installing the service manually. 2. If you run "exe" installer for Tomcat 7, it will log the commands that it used to install the service to a file in CATALINA_HOME/logs. BTW, official docs for the service wrapper itself are here: http://commons.apache.org/proper/commons-daemon/procrun.html >> >> Is path to Jvm a correct one? I think that is OK, but it could be >> simplified a bit. >> > > The path is dynamic and supplied by the installer. I merely climb up one > level to locate our dedicated copy of the JRE. > > If I subsequently run tomcat7w.exe, the JVM home is not set, nor are lots > of other matters. Just quick thoughts. 1. Are you running tomcat7w.exe as administrator? 2. Check list of services in Windows control panel. Is the service listed there? Check its properties and its command line there. 3. Check settings in the registry (see Commons Daemon procrun docs). 4. If you run tomcat7w.exe (prunmgr) without arguments then the file name is used as the service name. You used "Tomcat7" in IS command and the file is tomcat7w.exe. I think that is OK and that case sensivity does not matter here, as service.bat uses the same naming. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Inno Setup Script?
Getting quite frustrated now. Here's the line the installer is now running: C:\Program Files\Our Local Services\tomcat\bin\tomcat7.exe //IS//Tomcat7 --DisplayName="Apache Tomcat 7" --JavaHome="C:\Program Files\Our Local Services\tomcat\..\java-7-32bit\" --Install="C:\Program Files\Our Local Services\tomcat\bin\tomcat7.exe" --Jvm="C:\Program Files\Our Local Services\tomcat\..\java-7-32bit\bin\client\jvm.dll" --JvmOptions="-Djava.io.tmpdir=C:\Program Files\Our Local Services\tomcat\temp;-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager;-Djava.util.logging.config.file=C:\Program Files\Our Local Services\tomcat\conf\logging.properties;" --Classpath="C:\Program Files\Our Local Services\tomcat\bin\bootstrap.jar;C:\Program Files\Our Local Services\tomcat\bin\tomcat-juli.jar" --StartMode=jvm --StopMode=jvm --StartClass=org.apache.catalina.startup.Bootstrap --StartParams=start --StopClass=org.apache.catalina.startup.Bootstrap --StopParams=stop --LogPath="C:\Program Files\Our Local Services\tomcat\logs" --LogLevel=DEBUG If I subsequently run tomcat7w.exe, the JVM home is not set, nor are lots of other matters. Again, we're seeing error code 1 from Tomcat. EIther I'm doing something dumb or this just doesn't work as described. James
