Re: One process per webapp
Hi Tim, Tim Funk-2 wrote: 1) If relying on native - it might be easiest to place the System.loadLibrary() and its classes in the common classloader. Then use a Valve to call System.loadLibrary() so it's only called once. Hack, yes ... also very simple to do. This might allow webapps to be reloaded without a problem [depending on the nature of the change.] This won't work. I am modifying the native library in between reloads. Tim Funk-2 wrote: 2) apache httpd can easily handle the proxy scenario. Utilize the hot standby feature and have it point to the primary Tomcat. Then when you wish to restart ... you can bring up a new Tomcat which will be configured as the hot standby. Then shutdown the primary Tomcat and all traffic goes to the hot standby. Once the primary is down, change the proxy rules that now the failover is the primary and the standby will be located in the config for the next new tomcat instance. All the the proxy rules can be in an include. (a really simple one - probably less then 10 lines) You'd also need some httpd graceful restarts in there too to capture the new changes to the include proxy rule file. All of the this can probably be easily scriptable. (Even though there was a desire to avoid scripting) Great. The problem is that I need Netbeans to do all this legwork so that when I create a new web project I don't have to manually add all of this configuration. If this functionality was built into Tomcat, everyone work benefit automatically regardless of what IDE they use. Also note that it requires the use of Apache httpd which is even more stuff I'd have to configure/manage. Like I said from the beginning, this is more about ease-of-use and productivity than it is about a technical challenge. All of this is technically doable right now but it's not easy to do productively. Gili -- View this message in context: http://old.nabble.com/One-process-per-webapp-tp31836121p31851393.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: One process per webapp
Christopher Schultz-2 wrote: Why bother redeploying, then? Instead of doing ant redeploy, just do ant bounce or something similar. I am running inside Netbeans (remember the focus here is on improving development productivity, less so on improving production deployments). Modifying what they run under the hood is probably not going to be trivial. * Faster webapp redeploys if the entire JVM is restarted. I'm aiming for one second but currently we're at ten seconds. Christopher Schultz-2 wrote: Tomcat simply takes a certain amount of time to start. The fewer things you have configured at startup the better, but Tomcat starts up pretty quickly. Generally speaking, it's your webapp that is going to take a long time to start up if you have lots of things happening in ServletContextListeners, load-on-startup Servlets, etc. If you deploy no webapps, how long does it take Tomcat to start up? It takes about 10 seconds to shut down and start up a new instance. Starting up a new instance is fairly quick. Shutting down takes a while. When restarting a webapp, we don't have to wait for the old instance to finish shutting down (which is slow) before transferring access to another Tomcat instance. The new instance can simply bind to a different port and the proxy running at port 80 can switch which port it is redirecting requests to. The only snag I can think of is that the old instance might keep files locked on disk (but I think this is no longer the case with Tomcat 7.x) Gili -- View this message in context: http://old.nabble.com/One-process-per-webapp-tp31836121p31845924.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: One process per webapp
Pid * wrote: You can, right now, start blank Tomcat instances and deploy apps to them - managing the instance using JMX. Granted I can do a lot of this myself, but it's a lot of work for every user to reinvent this wheel. This kind of feature needs to be improved in the official release. Pid * wrote: What you can't do, is seamlessly transfer request processing from one running JVM to another - using Tomcat instances alone - because you can't exactly determine when one instance releases a port, enabling another to bind to it. You don't need to wait. You have a proxy webapp running on port 80 that redirects traffic to whatever port each webapp runs on. When an admin restarts a webapp you simply redirect traffic to the port of the new Tomcat instance (which may differ from the port of the instance you are shutting down). That way you don't need to wait for the old instance to shut down (which is slow) before launching the new one. Pid * wrote: Tomcat has some memory leak prevention capability (since 6.0.24) but JNI memory issues it can't prevent. What problems are you seeing? The new leak prevention is great, but last time I checked it couldn't solve 100% of offending libraries/webapps. Running webapps in their own JVM would be a 100% fix without having to resort to any magic/hacks. Gili -- View this message in context: http://old.nabble.com/One-process-per-webapp-tp31836121p31846007.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: One process per webapp
1. Always bounce Tomcat directly instead of re-loading the webapp
This is too slow (10-15 seconds) in its current form. If you use the
mechanism I'm proposing restart time is instant (you have a blank Tomcat
instance waiting to pick up a new webapp).
2. Fix the webapp so it doesn't bomb on startup when the library
is already loaded
As far as I know what you are proposing is impossible. Here is what I've
seen:
* A webapp must invoke System.loadLibrary() to use JNI.
* If a library is loaded by one ClassLoader and another ClassLoader invokes
System.loadLibrary() the JVM will throw: java.lang.UnsatisfiedLinkError:
Native Library x already loaded in another classloader.
* Tomcat loads each webapp into its own ClassLoader.
* When reloading a webapp, there is no guarantee that the old ClassLoader
will get garbage-collected (thereby ensuring that the old library is
unloaded) before the new webapp is launched. When I reload such webapps, I
get UnsatisfiedLinkError virtually 100% of the time.
* There is nothing a webapp can do to prevent this problem. Either each
webapp needs to run in its own JVM or Tomcat needs to guarantee that the old
ClassLoader is GCed before loading the new webapp instance.
Tomcat provides the manager webapp and ant tasks to access it, plus a
toolbox of scripts to start/stop/etc. Tomcat. Your needs seem to be
fairly specific... why not just roll your own solution?
Because then every user would have to roll their own solution. Right now it
is impossible to reload webapps that use JNI and it is extremely slow to
reload the entire JVM. Fixing this in the official release would benefit
anyone wishing to use JNI (or libraries that use JNI) from within Tomcat.
Nowadays with the rise of RESTful interfaces, it makes a lot more sense to
wrap a RESTful interface around libraries that interface with native code.
Christopher Schultz-2 wrote:
It sounds like the OP could just wrap the Tomcat launcher in a script
that looks roughly like this:
#/bin/sh
while [ 1 ]
do
${CATALINA_HOME}/bin/startup.sh run $CATALINA_OUT 21
done
This would start a Tomcat and wait for it to finish, then restart when
the original JVM came down. I suspect using JMX or whatever, one can
request a Tomcat shutdown which will ultimately terminate the JVM
(unless some foolish person launched a non-daemon thread at some point).
The script will re-start Tomcat.
This just doesn't seem like a difficult thing to implement oneself, nor
does it seem like something that is widely applicable.
Chris, I'm asking for a lot more sophistication than simply restarting the
JVM. I'm talking about sticking a proxy (on port 80) that acts a front-man.
Then each webapp run in its own JVM in some custom port. The proxy redirects
traffic to the custom ports. Reloading webapps would do the following:
* Grab a blank (loaded without any webapps) Tomcat instance from some pool,
ask it to load the new webapp instance.
* The new instance runs in a different port than the existing instance
* We ask the proxy to forward traffic to the new port instead of the old one
* We ask the old instance to shut down
* Note that traffic moves across instances instantly (without waiting for
the old instance to shut down)
I wouldn't want to try implementing this in terms of simple script files.
It's a lot more involved.
Gili
--
View this message in context:
http://old.nabble.com/One-process-per-webapp-tp31836121p31846107.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: One process per webapp
Christopher Schultz-2 wrote: Can't Netbeans bounce the Tomcat JVM for you? Sure, but if we use the existing mechanism of waiting for for the existing instance to shut down before restarting it then it takes at least 20 seconds each time. I'm looking for a second or less. Christopher Schultz-2 wrote: If this is just in development, why not just wrap System.loadLibrary() in a try/catch block and ignore the exception you get when re-loading the native library? This just seems like it would be a lot of work for not much gain. Because once you get this exception (whether you ignore it or not) you can no longer invoke any of the native methods (they will throw their own exceptions). Essentially, the reload operation has failed. Christopher Schultz-2 wrote: With no webapps deployed at all? Wow, something is not right. Are you sure you have enough memory installed on your dev instance? Starting up a new instance is fairly quick. Shutting down takes a while. That seems strange... do you have a complicated shutdown procedure for your webapp? Tomcat shuts down on my end in very short order when no webapps need to be undeployed. I don't think you and I are measuring the same thing. Tomcat loads up in under 2 seconds, but it takes over 8 seconds to shut down. It is quite possible that the Netbeans plugin is injecting a sleep() somewhere but I'm not sure. I know they have a 5 second sleep for the Glassfish plugin. I'm guessing they had a good reason for adding those calls though. Does Tomcat provide hooks for integrators to know that the webapp is fully unloaded? How long does a restart (without a webapp) take on your end? Christopher Schultz-2 wrote: When restarting a webapp, we don't have to wait for the old instance to finish shutting down (which is slow) before transferring access to another Tomcat instance. If they're going to be using the same port, you do. They wouldn't use the same port. Christopher Schultz-2 wrote: The new instance can simply bind to a different port and the proxy running at port 80 can switch which port it is redirecting requests to. So... you want Tomcat to also notify a 3rd-party that it's being shut down so that a proxy can re-configure itself? No. I would ask the proxy (which would be part of Tomcat) to restart an instance on my behalf. It would do the necessary communication with the instance to request a shutdown and listen for lifecycle events. Christopher Schultz-2 wrote: The only snag I can think of is that the old instance might keep files locked on disk (but I think this is no longer the case with Tomcat 7.x) I don't believe Tomcat locks anything specific. Under certain situations, the JVM and the OS conspire to lock certain resources (usually .jar files) and that can be a problem. If you're starting a new instance in parallel, wouldn't you want to do it from a separate directory anyway? Good point. We could do that. Gili -- View this message in context: http://old.nabble.com/One-process-per-webapp-tp31836121p31847022.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: One process per webapp
Christopher Schultz-2 wrote: So, now you want Tomcat to launch multiple copies of itself as hot standbys for /potential future webapp deployments/? We'll never do this. The exact mechanism is not fixed in stone (I'm sure we could improve on the initial idea). One approach is that administrators should be able to configure webapps in development mode, and in only such a case you'd launch a hot standby. Christopher Schultz-2 wrote: * A webapp must invoke System.loadLibrary() to use JNI. Must it? You can load a native library at a higher-level instead of in the webapp itself and avoid all of this. That's not good enough. Webapps that use JNI will modify DLLs between reloads. Even if you If you load the DLL in some parent ClassLoader you'll still need to unload it every time the webapp is reloaded, otherwise you'll miss changes to the DLL. Christopher Schultz-2 wrote: * If a library is loaded by one ClassLoader and another ClassLoader invokes System.loadLibrary() the JVM will throw: java.lang.UnsatisfiedLinkError: Native Library x already loaded in another classloader. I don't believe this has anything to do with ClassLoaders. Yes it does. Read the evaluation of http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4286309 You may only load a library into one ClassLoader at a time. The library is only unloaded from the JVM when the ClassLoader is garbage-collected. Christopher Schultz-2 wrote: I don't think the ClassLoader ever unloads the native library. I haven't found any documentation that suggests that it will. This was indeed hard to track down. See the evaluation of http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4119554 and http://codethesis.com/tutorial.php?id=1 Worse-case scenario we could run independent tests to verify this but either way I think we can both agree the safest and easiest way to ensure proper reloading of native libraries is to restart the JVM. Christopher Schultz-2 wrote: Or you can put your .jar that wraps your native library in Tomcat's lib directory and take it out of your webapp. That's not the most portable solution, but neither is using a native library in the first place. That's not an option because as I've mentioned before the native library is potentially the very thing you are redeploying. Christopher Schultz-2 wrote: It's not a good excuse, but native libraries are a pain in the ass in Java. If you use one, be prepared to have your ass hurt. :) With all due respect, they are only painful to use in web containers. They are extremely simple to use in desktop applications. Christopher Schultz-2 wrote: Aah, you are a REST guy. That explains the constant appeals against practicality. ;) If you wrapped your JNI library with a REST interface, couldn't you avoid all this foolishness? That's precisely what I'm doing :) I still need to be able to redeploy the native library though. Think of the native library + RESTful interface + Tomcat as an appliance. I'm making regular updates to the library and I need to restart it quickly to speed up the debugging process. Gili -- View this message in context: http://old.nabble.com/One-process-per-webapp-tp31836121p31847177.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
One process per webapp
Hi, I posted a RFE at https://issues.apache.org/bugzilla/show_bug.cgi?id=51366 asking for the ability to seamlessly deploy webapps into separate JVMs. Tomcat 7.0's parallel deployment sounds nice but it still doesn't solve the JNI and memory leak problems that haunt a single JVM architecture. Please read the proposal and let me know what you think. Thanks, Gili -- View this message in context: http://old.nabble.com/One-process-per-webapp-tp31836121p31836121.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: One process per webapp
Hi Christopher, Assuming we simply use one webapp per Tomcat instance I'm still looking for two changes: * Restart the entire JVM on webapp redeploy (to avoid JNI and ClassLoader problems) * Faster webapp redeploys if the entire JVM is restarted. I'm aiming for one second but currently we're at ten seconds. Gili Christopher Schultz-2 wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gili, On 6/13/2011 1:07 PM, cowwoc wrote: I posted a RFE at https://issues.apache.org/bugzilla/show_bug.cgi?id=51366 asking for the ability to seamlessly deploy webapps into separate JVMs. So you want Tomcat to have an option to run as a supervisor in one JVM and deploy webapps to separate JVMs? Tomcat 7.0's parallel deployment sounds nice but it still doesn't solve the JNI and memory leak problems that haunt a single JVM architecture. Parallel deployment is at once orthogonal to and the opposite of what you are requesting. Please read the proposal and let me know what you think. What single management interface are you describing in your enhancement comments? The Tomcat manager webapp? It's trivial to run a manager in each JVM and use that for deployment. If you know that your webapp needs to do things such as register a shared library on startup, you can do one of two things: 1. Always bounce Tomcat directly instead of re-loading the webapp 2. Fix the webapp so it doesn't bomb on startup when the library is already loaded Tomcat provides the manager webapp and ant tasks to access it, plus a toolbox of scripts to start/stop/etc. Tomcat. Your needs seem to be fairly specific... why not just roll your own solution? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk32htcACgkQ9CaO5/Lv0PAGJwCaA020eDYhHSRmgVxALLLisG7z 0zgAn1WDZNd+BRnJVD+C/kKUHh6KtotJ =o0Md -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- View this message in context: http://old.nabble.com/One-process-per-webapp-tp31836121p31839136.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
