Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test
Looks like attachments are not allowed so only sharing the contents of threadDump3.out 2015-11-04 22:55:58 Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode): "http-nio-8443-exec-147" daemon prio=10 tid=0x7f5c1c005000 nid=0x6241 runnable [0x7f5bd30ef000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) - locked <0xfc2d9498> (a java.lang.Object) at org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:443) at org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:436) at org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:795) at org.apache.coyote.http11.InternalNioInputBuffer.parseRequestLine(InternalNioInputBuffer.java:227) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:993) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1760) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1719) - locked <0x875077f0> (a org.apache.tomcat.util.net.SecureNioChannel) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x88a613f0> (a java.util.concurrent.ThreadPoolExecutor$Worker) "http-nio-8443-exec-146" daemon prio=10 tid=0x7f5c18020800 nid=0x6240 waiting on condition [0x7f5c6e271000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x86b7a088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "http-nio-8443-exec-145" daemon prio=10 tid=0x7f5c18037000 nid=0x623f waiting on condition [0x7f5c6e372000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x86b7a088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "http-nio-8443-exec-144" daemon prio=10 tid=0x7f5c1802f800 nid=0x623e waiting on condition [0x7f5c6c89c000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x86b7a088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.c
Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test
Hi Chris, I captured 3 threadumps with the below requests/second when the server is already slow. We were using BIO before and had observed slowness even with that. We switched to using NIO since it seemed a recommendation for production environment. -- *WITH NIO* *--* *RUN1 (threadDump1.out)* Requests/second = 18849.1606557377 *RUN2 **(threadDump2.out)* Requests/second = 18943.43606557377 *RUN3 **(threadDump3.out)* Requests/second = 18894.21241830065 -- The issue we are trying to address why does performance degrade by 50% with subsequent runs. Thanks, Dimple On Thu, Nov 5, 2015 at 1:31 AM, Christopher Schultz < ch...@christopherschultz.net> wrote: > Dimple, > > On 11/3/15 11:24 PM, dimple ranka wrote: > > Also forgot to mention that setting maxKeepAliveRequests to -1 did not > > help. As expected from the connector documentation the default value for > > this attribute is 100 and we have a 60 user test set up. > > > > On Wed, Nov 4, 2015 at 8:18 AM, dimple ranka > wrote: > > > >> Hi Mark, > >> > >> Another observation to be noted here is that system CPU usage shoots up > >> for subsequent runs, especially for later runs. > >> > >> We have been looking into this issue for couple of weeks now and it is > >> clear in the same environment with the same setup it doesn't reproduce > on > >> tomcat6. The moment we deploy the web application in a tomcat7 container > >> the slowness with subsequent runs shows up. > > Can you take some thread dumps to find out what Tomcat is doing? High > CPU usage and lower request throughput sounds like a Poller problem > since you are using NIO. > > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Can adding workers to worker.list and balance_workers cause an issue?
What would happen if I add workers into worker.list and balance_workers? Option1: worker.list=node1,node2,loadbalancer,status worker.loadbalancer.balance_workers=node1,node2 VS Option2: worker.list=loadbalancer,status worker.loadbalancer.balance_workers=node1,node2 If I went with Option1 will that cause errors with the apache LB like cause blank page loads or perhaps slow down performance? What I am looking for is, what is the repercussions with going with Option1? Thank You, Jonathan Fonken IT Sys Admin Office: (605) 508-4359 jonathan.fon...@xtime.com 1400 Bridge Parkway, Suite 200 Redwood City, CA 94065 www.xtime.com
Re: Trouble downloading large file (>1GB)
Jamie, On 11/4/15 9:20 AM, jamie wrote: > We are using Apache Tomcat Version 7.0.59. We have received downloads > from our servlet greater than 1GB in size complete prematurely. Smaller > files are no problem. All data is streamed so no OME occurs. OME? > We tried increasing the connection timeout on the connector, but it > made no difference. The connection timeout here isn't relevant. > This looks to be a Tomcat related issue. There is no any > specific error reported in the Tomcat logs. The download completes short > of the actual file size. Nothing at all in the logs? > The connector is defined as follows: > > protocol="org.apache.coyote.http11.Http11NioProtocol" > redirectPort="8443" URIEncoding="UTF-8" compression="on" > noCompressionUserAgents="gozilla, traviata" > compressableMimeType="text/html,text/xml,text/css,text/plain,application/javascript,application/json" > enableLookups="false" disableUploadTimeout="true" /> > > Any ideas, on what might be the cause? What code on the server side is serving this resource? Is it the DefaultServlet (static file) or some code you wrote? What happens if you disable compression? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test
Dimple, On 11/3/15 11:24 PM, dimple ranka wrote: > Also forgot to mention that setting maxKeepAliveRequests to -1 did not > help. As expected from the connector documentation the default value for > this attribute is 100 and we have a 60 user test set up. > > On Wed, Nov 4, 2015 at 8:18 AM, dimple ranka wrote: > >> Hi Mark, >> >> Another observation to be noted here is that system CPU usage shoots up >> for subsequent runs, especially for later runs. >> >> We have been looking into this issue for couple of weeks now and it is >> clear in the same environment with the same setup it doesn't reproduce on >> tomcat6. The moment we deploy the web application in a tomcat7 container >> the slowness with subsequent runs shows up. Can you take some thread dumps to find out what Tomcat is doing? High CPU usage and lower request throughput sounds like a Poller problem since you are using NIO. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Making fail-over work with PermGen errors
Stephen, On 11/3/15 4:01 PM, Stephen Booth wrote: > Does anyone have any advice about how to make fail-over work more > cleanly with PermGen errors (other than the obvious one of not letting > PermGen errors occur) > > I'm using apache ModProxy to connect to tomcat over ajp with additional > hot-spare servers confgured for fail-over but if I mess up and let the > jvm run out of PermGen space then tomcat seems to stay alive enough to > prevent fail-over but not enough to actually serve any content. Memory-related errors are always problematic. Do you have any instrumentation/monitoring to check the liveness of your Tomcat nodes? Can it detect the OOME condition? If so, you should be able to notify the load-balancer that something had gone wrong. I'm unsure of how to do it with mod_proxy_ajp, but mod_jk can be re-configured externally e.g. by calling curl with a specially-crafted URL[1]. -chris [1] https://wiki.apache.org/tomcat/tools/mod_jk.py - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] RE: 80ms delay switching between worker threads
Farzad, On 11/3/15 2:45 PM, Farzad Panahi wrote: > I wish I could get my hands on a real server : ) What computer do you use to access the server? That one will certainly work for this kind of testing. -chris > On Mon, Nov 2, 2015 at 12:23 PM, David kerber wrote: >> On 11/2/2015 3:09 PM, Farzad Panahi wrote: >>> >>> Quoting from David Holme's blog: >>> The nanoTime method uses the highest resolution clock available on the platform, and while its return value is in nanoseconds, the update resolution is typically only microseconds. >>> >>> https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks >>> >>> I think we can rely on nanoTime as a clock with microsecond >>> resolution. Having said that can't we say printing out nanoTime in >>> websocket message handler will give us a fair number (with microsecond >>> accuracy) to measure how quickly the message handler is being called? >>> >>> All I am saying is that I see an obvious hiccup in order of >>> milliseconds when threads are switching which I have no explanation >>> for. >>> >>> Please advise if you think the way I am measuring is wrong. >> >> >> I'm with Chris on this one: I think it's due to running on a VM rather than >> on real hardware. >> >> >> >>> >>> Cheers >>> >>> Farzad >>> >>> On Mon, Nov 2, 2015 at 4:56 AM, David kerber wrote: On 10/31/2015 10:51 AM, David Balažic wrote: > > > Just a note: When most of you say "resolution" what you think about is > actually called "accuracy". > (also see "precision" , here is a good roundup: > http://www.tutelman.com/golf/measure/precision.php ) I'm not sure about the others, but as an Electrical Engineer, I know the difference between resolution, precision, and accuracy. In the post I made earlier, I said and meant "resolution". > > David Balažic > Software Engineer > www.comtrade.com > >> -Original Message- >> From: Konstantin Preißer [mailto:kpreis...@apache.org] >> Sent: 31. October 2015 10:27 >> To: Tomcat Users List >> Subject: [OT] RE: 80ms delay switching between worker threads >> Importance: Low >> >> Hi Christopher, >> >>> -Original Message- >>> From: Christopher Schultz [mailto:ch...@christopherschultz.net] >>> Sent: Saturday, October 31, 2015 3:43 AM >>> >>> What OS are you using? IIRC, the Windows timer has horrible >>> resolution. >>> you can call System.currentTimeNanos all you want, but you won't get >>> anything meaningful lower than some threshold regardless of the actual >>> least significant digits coming back from those calls. >> >> >> >> While that may have been true in ancient versions like XP and Vista, at >> least >> starting with Win7 QueryPerformanceCounter() uses the processor's TSC >> [1] >> (where Vista used the HPET if available) so you should have a very high >> resolution here. E.g. running the following Java program: >> >> int[] iterations = { 100, 120, 150, 250 }; >> >> for (int i = 0; i < iterations.length; i++) { >> for (int j = 0; j < 3; j++) { >> long currentTime = System.nanoTime(); >> double startValue = 1000; >> for (int z = 0; z < iterations[i]; z++) { >> startValue = Math.pow(startValue, 0.99); >> } >> long difference = System.nanoTime() - currentTime; >> System.out.println(iterations[i] + " pow iterations ms >> took >> " + >> (difference / 1000L) + " µs"); >> } >> } >> >> prints on my system something like: >> >> 100 pow iterations ms took 25 µs >> 100 pow iterations ms took 7 µs >> 100 pow iterations ms took 7 µs >> 120 pow iterations ms took 8 µs >> 120 pow iterations ms took 9 µs >> 120 pow iterations ms took 8 µs >> 150 pow iterations ms took 11 µs >> 150 pow iterations ms took 10 µs >> 150 pow iterations ms took 13 µs >> 250 pow iterations ms took 18 µs >> 250 pow iterations ms took 17 µs >> 250 pow iterations ms took 17 µs >> >> >> So there should at least be a microsecond resolution. On a C# program >> using >> Stopwatch I get similar results in the range from 5 to 12 µs. >> >> Note, QueryPerformanceFrequency() [2] can be used to get the frequency >> of the timer which is exposed in .Net through static >> System.Diagnostics.Stopwatch.Frequency field as ticks per second. On my >> system it prints "3323580" so the resolution should be around ~0.3 >> microseconds. >> >> >> Regards, >> Konstantin Preißer >> >> [1] https://msdn.microsoft.com/en- >> us/library/windows/desktop/dn553408%28v=vs.85%29.aspx >> [2] https://msdn.microsoft.com/de- >> de/library/windows/
Re: port 80
2015-11-04 20:20 GMT+03:00 Linux Support : > Hi again, > > configured the TC service to run as a non privileged user. In my > understanding we cannot use a privileged port to bind TC to. Is there a way > i can use port 80 for TC in the case of using a non root user ? http://tomcat.apache.org/ -> FAQ -> How To -> "How to run Tomcat without root privileges?" - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: port 80
If you're running on a UNIX variant, use JSVC. The source is included in the tomcat download in bin/commons-daemon/native.tar.gz. On 11/4/2015 10:20 AM, Linux Support wrote: Hi again, configured the TC service to run as a non privileged user. In my understanding we cannot use a privileged port to bind TC to. Is there a way i can use port 80 for TC in the case of using a non root user ? cheers osp -- George Sexton *MH Software, Inc.* Voice: 303 438 9585 http://www.mhsoftware.com
port 80
Hi again, configured the TC service to run as a non privileged user. In my understanding we cannot use a privileged port to bind TC to. Is there a way i can use port 80 for TC in the case of using a non root user ? cheers osp
Re: Newbie tomcat 8.0.28 question
Thanks all. I will incorporate the digested passwd On Wed, Nov 4, 2015 at 2:10 AM, Christopher Schultz < ch...@christopherschultz.net> wrote: > To whom it may concern, > > On 11/3/15 6:39 AM, Linux Support wrote: > > not having a clear text password in the users.xml ( i think ) for the > admin > > user. My understanding is that the admin user logging in through the > > default page can do a deployment. > > Do you mean like this? > > http://tomcat.apache.org/tomcat-8.0-doc/realm-howto.html#Digested_Passwords > > (That documentation is a little off... you can use password-munging > algorithms that are not provided by the MessageDigest class). > > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Monitoring Connections
By the way, I had to back-burner this investigation due to other things that have come up. When those are out of the way, I'll pick up where I left off. Thanks for the discussion so far. (I don't know anything about SNMP, etc., so I'm only following that thread very loosely until I have time to concentrate on it.) On Wed, Nov 4, 2015 at 3:25 AM, Aurélien Terrestris wrote: > Hi Chris > > I still recommend SNMP, maybe I expressed myself incorrectly but Java SNMP > support is made for monitoring. I also provided a bad link, here is the > good one : > http://docs.oracle.com/javase/7/docs/technotes/guides/management/snmp.html > > When using the right server, the SNMP will also provide alerting, both for > counters and deltas. This is the critical point for large scale teams with > N1 support, although developers often want something else and need more > (logs, instrumentation). > > In "Effective Monitoring and Alerting, O'Reilly" we find these definitions > that we already agree, I believe : > > "The main purpose of monitoring is to gain near real-time insight into the > current state of the system, in the context of its recent performance" > > "The core functionality of an alarm is to trigger on detection of abnormal > timeseries behavior, but the alerting system should also support the > aggregation and conditional suppression of alarms." > > In my opinion, SNMP could be more used, and I notice that nobody is talking > about this technology for years on the mailing-list. It's a pity to ignore > solutions. > > > best regards > > > > > 2015-11-03 16:05 GMT+01:00 Christopher Schultz < > ch...@christopherschultz.net > >: > > > Aurelien, > > > > On 11/2/15 5:54 PM, Aurélien Terrestris wrote: > > > Either my reply was not read, or I'm surprised nobody is answering > here. > > > > > > "1. Java doesn't directly support SNMP;" > > > > > > It does, officially : > > http://docs.oracle.com/javase/7/docs/technotes/guides/ > > > management/snmp.html and it's working pretty well with Tomcat besides > > being > > > easy to set up. > > > > I meant there was no client API > > > > > "2. If the JVM is braindead, the SNMP agent can't announce any state > to > > > the managers." > > > > > > Agent ? This is the SNMP server which is polling the Tomcat here. When > > > Tomcat is dead, the server cannot poll anymore and raises an alert. > > > > You were recommending the use of SNMP as an "alerting" alternative to > > "monitoring". I assumed you had meant that more passive "alerting" (e.g. > > only take action when a problem is occurring) was more efficient than > > "monitoring" (which I took to mean polling a service to check its > status). > > > > So polling is less efficient but more reliable. Whether you use SNMP, > > JMX, HTTP, or whatever, *polling* a component is better than having a > > component try to report that it's failing, because part of the failure > > could be its inability to report the failure state. > > > > This isn't about SNMP versus some other technology. It's about probing > > versus ... not doing so, I guess. > > > > > You see how alerting is even easier without instrumenting, but maybe > you > > > want to go deeper. But then, you probably start mixing what's necessary > > for > > > N1 support with what's helpful for developers. They have different > jobs. > > > > What is your definition of "instrumentation"? What about "probing" or > > "alerting"? I'm certainly confused with your use of terms, especially > > when you equate one of them with a particular technology. > > > > -chris > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > >
Trouble downloading large file (>1GB)
Greetings We are using Apache Tomcat Version 7.0.59. We have received downloads from our servlet greater than 1GB in size complete prematurely. Smaller files are no problem. All data is streamed so no OME occurs. We tried increasing the connection timeout on the connector, but it made no difference. This looks to be a Tomcat related issue. There is no any specific error reported in the Tomcat logs. The download completes short of the actual file size. The connector is defined as follows: protocol="org.apache.coyote.http11.Http11NioProtocol" redirectPort="8443" URIEncoding="UTF-8" compression="on" noCompressionUserAgents="gozilla, traviata" compressableMimeType="text/html,text/xml,text/css,text/plain,application/javascript,application/json" enableLookups="false" disableUploadTimeout="true" /> Any ideas, on what might be the cause? Jamie - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Monitoring Connections
Hi Chris I still recommend SNMP, maybe I expressed myself incorrectly but Java SNMP support is made for monitoring. I also provided a bad link, here is the good one : http://docs.oracle.com/javase/7/docs/technotes/guides/management/snmp.html When using the right server, the SNMP will also provide alerting, both for counters and deltas. This is the critical point for large scale teams with N1 support, although developers often want something else and need more (logs, instrumentation). In "Effective Monitoring and Alerting, O'Reilly" we find these definitions that we already agree, I believe : "The main purpose of monitoring is to gain near real-time insight into the current state of the system, in the context of its recent performance" "The core functionality of an alarm is to trigger on detection of abnormal timeseries behavior, but the alerting system should also support the aggregation and conditional suppression of alarms." In my opinion, SNMP could be more used, and I notice that nobody is talking about this technology for years on the mailing-list. It's a pity to ignore solutions. best regards 2015-11-03 16:05 GMT+01:00 Christopher Schultz : > Aurelien, > > On 11/2/15 5:54 PM, Aurélien Terrestris wrote: > > Either my reply was not read, or I'm surprised nobody is answering here. > > > > "1. Java doesn't directly support SNMP;" > > > > It does, officially : > http://docs.oracle.com/javase/7/docs/technotes/guides/ > > management/snmp.html and it's working pretty well with Tomcat besides > being > > easy to set up. > > I meant there was no client API > > > "2. If the JVM is braindead, the SNMP agent can't announce any state to > > the managers." > > > > Agent ? This is the SNMP server which is polling the Tomcat here. When > > Tomcat is dead, the server cannot poll anymore and raises an alert. > > You were recommending the use of SNMP as an "alerting" alternative to > "monitoring". I assumed you had meant that more passive "alerting" (e.g. > only take action when a problem is occurring) was more efficient than > "monitoring" (which I took to mean polling a service to check its status). > > So polling is less efficient but more reliable. Whether you use SNMP, > JMX, HTTP, or whatever, *polling* a component is better than having a > component try to report that it's failing, because part of the failure > could be its inability to report the failure state. > > This isn't about SNMP versus some other technology. It's about probing > versus ... not doing so, I guess. > > > You see how alerting is even easier without instrumenting, but maybe you > > want to go deeper. But then, you probably start mixing what's necessary > for > > N1 support with what's helpful for developers. They have different jobs. > > What is your definition of "instrumentation"? What about "probing" or > "alerting"? I'm certainly confused with your use of terms, especially > when you equate one of them with a particular technology. > > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >