Re: Performance problems (More Info).
SMP = Symmetric Multiprocessing. It's a multi-CPU box. Jeff. On Sat, 07 Apr 2001 14:02:37 +0200 "Marco Pas (GMX)" <[EMAIL PROTECTED]> wrote: >Sorry for my stupid question, but what is a >a SMP machine ? > >Kind Regards, >Marco > >At 18:13 2-4-01 -0700, you wrote: >>Well I found the problem with this and I thought I'd let everyone know for >>the sake of posterity. >> >>I found that this problem did not occur on my non-SMP machines, so I played >>around and found that the HotSpot server behaves wierd with SMP. Switching >>to pure interpretted mode fixes the problem. >> >>It wasn't Orion... >> >>-Original Message- >>From: Aaron Tavistock [mailto:[EMAIL PROTECTED]] >>Sent: Monday, March 26, 2001 12:11 PM >>To: Orion-Interest >>Subject: RE: Performance problems (More Info). >> >> >>JVM Settings -- My normal settings are "-server -Xincgc -Xms128m -Xmx384m". >>I've tried playing around with different JVM options including turning off >>server, using normal gc, running in interpreted mode, running in classic >>mode, et al. I even tried running under JDK1.2.2. While these definately >>effect performance in dramatic ways, none seems to fix this problem. >> >>Garbage Collection -- I tried watching the heap by turning on verbosegc, and >>the pauses do not appear to be timed with a full gc. So thats not it >>either, definately a good thought though... >> >>DB Server -- The DB server itself is performing fine, CPU and memory >>utilization is low. The database is completely accessable via sqlplus or an >>alternative app while these pauses occur. >> >>Connection Pooling vs Orions DataSource Manager -- Interestingly the >>problem seems to go away when I turn off Orions datasources, and run with a >>third party ConnectionPool manager (such as BitMechanic).While this >>fixes my immediate problem it will prevent me from using Orion as an EJB >>container, which is not good. >> >> > -Original Message- >> > From: [EMAIL PROTECTED] >> > [mailto:[EMAIL PROTECTED]]On Behalf Of Aaron >> > Tavistock >> > Sent: Monday, March 26, 2001 1:09 PM >> > To: Orion-Interest >> > Subject: Performance problems... >> > >> > >> > I've been working on getting Orion running in a production >> > environment for a >> > little while now and just when I thought everything was working >> > fine I go to >> > push to production and something load/volume related is creating massive >> > slowdowns. >> > >> > Basically every 250 database accesses or so there is a long pause >> > (20 to 60 >> > second), where nothing occurs. During this pause the CPU load *drops* to >> > practically nothing and our entire site is frozen. I'm not sure exactly >> > where the problem exists; it could be our code, our >> > configuration, or even a >> > bug in Orion. >> > >> > The environment is Redhat 6.2, JDK1.3, Oracle 8i. Its a pure >> > J2EE app, but >> > we're not using EJB. I initially thought it might be a memory issue, but >> > I've played with the JDK heap size and carefully watched memory >> > utilization >> > and thats also not the issue. I even considered that maybe >> > Evermind/IronFlare might have a throttle (to push you to get a >> > license) so I >> > put one of our production licenses on the QA box. >> > >> > I've since gotten a load tester and can reproduce the problem. Oddly, it >> > only happens on pages which require database access. Even more >> > interesting >> > is that it occurs more frequently on pages which utilize more than one >> > connection. But thats about as far as I can narrow it. I've tried the >> > 8.15 and 8.17 type4 jdbc drivers from oracle and we've tried Oracles >> > ConnectionCacheImpl and Orions XADataSource implimentation, both show the >> > same behavior (though both are using the Oracle Driver). I've also tried >> > Orions jdbc debug and it shows nothing of interest. >> > >> > So far I've put about a week straight into finding it, and I've just about >> > run out of ideas. I'd really be appreciative if anyone has any good >> > suggestions on where to look. ANyone seen behavior like this before? >> > >> > >
RE: Performance problems (More Info).
Sorry for my stupid question, but what is a a SMP machine ? Kind Regards, Marco At 18:13 2-4-01 -0700, you wrote: >Well I found the problem with this and I thought I'd let everyone know for >the sake of posterity. > >I found that this problem did not occur on my non-SMP machines, so I played >around and found that the HotSpot server behaves wierd with SMP. Switching >to pure interpretted mode fixes the problem. > >It wasn't Orion... > >-Original Message- >From: Aaron Tavistock [mailto:[EMAIL PROTECTED]] >Sent: Monday, March 26, 2001 12:11 PM >To: Orion-Interest >Subject: RE: Performance problems (More Info). > > >JVM Settings -- My normal settings are "-server -Xincgc -Xms128m -Xmx384m". >I've tried playing around with different JVM options including turning off >server, using normal gc, running in interpreted mode, running in classic >mode, et al. I even tried running under JDK1.2.2. While these definately >effect performance in dramatic ways, none seems to fix this problem. > >Garbage Collection -- I tried watching the heap by turning on verbosegc, and >the pauses do not appear to be timed with a full gc. So thats not it >either, definately a good thought though... > >DB Server -- The DB server itself is performing fine, CPU and memory >utilization is low. The database is completely accessable via sqlplus or an >alternative app while these pauses occur. > >Connection Pooling vs Orions DataSource Manager -- Interestingly the >problem seems to go away when I turn off Orions datasources, and run with a >third party ConnectionPool manager (such as BitMechanic).While this >fixes my immediate problem it will prevent me from using Orion as an EJB >container, which is not good. > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED]]On Behalf Of Aaron > > Tavistock > > Sent: Monday, March 26, 2001 1:09 PM > > To: Orion-Interest > > Subject: Performance problems... > > > > > > I've been working on getting Orion running in a production > > environment for a > > little while now and just when I thought everything was working > > fine I go to > > push to production and something load/volume related is creating massive > > slowdowns. > > > > Basically every 250 database accesses or so there is a long pause > > (20 to 60 > > second), where nothing occurs. During this pause the CPU load *drops* to > > practically nothing and our entire site is frozen. I'm not sure exactly > > where the problem exists; it could be our code, our > > configuration, or even a > > bug in Orion. > > > > The environment is Redhat 6.2, JDK1.3, Oracle 8i. Its a pure > > J2EE app, but > > we're not using EJB. I initially thought it might be a memory issue, but > > I've played with the JDK heap size and carefully watched memory > > utilization > > and thats also not the issue. I even considered that maybe > > Evermind/IronFlare might have a throttle (to push you to get a > > license) so I > > put one of our production licenses on the QA box. > > > > I've since gotten a load tester and can reproduce the problem. Oddly, it > > only happens on pages which require database access. Even more > > interesting > > is that it occurs more frequently on pages which utilize more than one > > connection. But thats about as far as I can narrow it. I've tried the > > 8.15 and 8.17 type4 jdbc drivers from oracle and we've tried Oracles > > ConnectionCacheImpl and Orions XADataSource implimentation, both show the > > same behavior (though both are using the Oracle Driver). I've also tried > > Orions jdbc debug and it shows nothing of interest. > > > > So far I've put about a week straight into finding it, and I've just about > > run out of ideas. I'd really be appreciative if anyone has any good > > suggestions on where to look. ANyone seen behavior like this before? > > > >
RE: Performance problems (More Info).
Well I found the problem with this and I thought I'd let everyone know for the sake of posterity. I found that this problem did not occur on my non-SMP machines, so I played around and found that the HotSpot server behaves wierd with SMP. Switching to pure interpretted mode fixes the problem. It wasn't Orion... -Original Message- From: Aaron Tavistock [mailto:[EMAIL PROTECTED]] Sent: Monday, March 26, 2001 12:11 PM To: Orion-Interest Subject: RE: Performance problems (More Info). JVM Settings -- My normal settings are "-server -Xincgc -Xms128m -Xmx384m". I've tried playing around with different JVM options including turning off server, using normal gc, running in interpreted mode, running in classic mode, et al. I even tried running under JDK1.2.2. While these definately effect performance in dramatic ways, none seems to fix this problem. Garbage Collection -- I tried watching the heap by turning on verbosegc, and the pauses do not appear to be timed with a full gc. So thats not it either, definately a good thought though... DB Server -- The DB server itself is performing fine, CPU and memory utilization is low. The database is completely accessable via sqlplus or an alternative app while these pauses occur. Connection Pooling vs Orions DataSource Manager -- Interestingly the problem seems to go away when I turn off Orions datasources, and run with a third party ConnectionPool manager (such as BitMechanic).While this fixes my immediate problem it will prevent me from using Orion as an EJB container, which is not good. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]]On Behalf Of Aaron > Tavistock > Sent: Monday, March 26, 2001 1:09 PM > To: Orion-Interest > Subject: Performance problems... > > > I've been working on getting Orion running in a production > environment for a > little while now and just when I thought everything was working > fine I go to > push to production and something load/volume related is creating massive > slowdowns. > > Basically every 250 database accesses or so there is a long pause > (20 to 60 > second), where nothing occurs. During this pause the CPU load *drops* to > practically nothing and our entire site is frozen. I'm not sure exactly > where the problem exists; it could be our code, our > configuration, or even a > bug in Orion. > > The environment is Redhat 6.2, JDK1.3, Oracle 8i. Its a pure > J2EE app, but > we're not using EJB. I initially thought it might be a memory issue, but > I've played with the JDK heap size and carefully watched memory > utilization > and thats also not the issue. I even considered that maybe > Evermind/IronFlare might have a throttle (to push you to get a > license) so I > put one of our production licenses on the QA box. > > I've since gotten a load tester and can reproduce the problem. Oddly, it > only happens on pages which require database access. Even more > interesting > is that it occurs more frequently on pages which utilize more than one > connection. But thats about as far as I can narrow it. I've tried the > 8.15 and 8.17 type4 jdbc drivers from oracle and we've tried Oracles > ConnectionCacheImpl and Orions XADataSource implimentation, both show the > same behavior (though both are using the Oracle Driver). I've also tried > Orions jdbc debug and it shows nothing of interest. > > So far I've put about a week straight into finding it, and I've just about > run out of ideas. I'd really be appreciative if anyone has any good > suggestions on where to look. ANyone seen behavior like this before? > >
RE: Performance problems (More Info).
Aaron Tavistock ([EMAIL PROTECTED]) wrote: > JVM Settings -- My normal settings are "-server -Xincgc -Xms128m -Xmx384m". > I've tried playing around with different JVM options including turning off > server, using normal gc, running in interpreted mode, running in classic > mode, et al. I even tried running under JDK1.2.2. While these definately > effect performance in dramatic ways, none seems to fix this problem. > > Garbage Collection -- I tried watching the heap by turning on verbosegc, and > the pauses do not appear to be timed with a full gc. So thats not it > either, definately a good thought though... > > DB Server -- The DB server itself is performing fine, CPU and memory > utilization is low. The database is completely accessable via sqlplus or an > alternative app while these pauses occur. > > Connection Pooling vs Orions DataSource Manager -- Interestingly the > problem seems to go away when I turn off Orions datasources, and run with a > third party ConnectionPool manager (such as BitMechanic).While this > fixes my immediate problem it will prevent me from using Orion as an EJB > container, which is not good. Have you tried using the com.evermind.sql.ConnectionDataSource? Frankly I have no idea what it is, since it's completely undocumented, but it was recommended to me by Orion support folk a year or so ago. I've been using it ever since. For what it's worth... Gary p.s. Here's my data-sources.xml. We've been using Orion exclusively as a web server so far (in production three months or so?), so the only connection we've used from the set below is 'pooled-location'. > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED]]On Behalf Of Aaron > > Tavistock > > Sent: Monday, March 26, 2001 1:09 PM > > To: Orion-Interest > > Subject: Performance problems... > > > > > > I've been working on getting Orion running in a production > > environment for a > > little while now and just when I thought everything was working > > fine I go to > > push to production and something load/volume related is creating massive > > slowdowns. > > > > Basically every 250 database accesses or so there is a long pause > > (20 to 60 > > second), where nothing occurs. During this pause the CPU load *drops* to > > practically nothing and our entire site is frozen. I'm not sure exactly > > where the problem exists; it could be our code, our > > configuration, or even a > > bug in Orion. > > > > The environment is Redhat 6.2, JDK1.3, Oracle 8i. Its a pure > > J2EE app, but > > we're not using EJB. I initially thought it might be a memory issue, but > > I've played with the JDK heap size and carefully watched memory > > utilization > > and thats also not the issue. I even considered that maybe > > Evermind/IronFlare might have a throttle (to push you to get a > > license) so I > > put one of our production licenses on the QA box. > > > > I've since gotten a load tester and can reproduce the problem. Oddly, it > > only happens on pages which require database access. Even more > > interesting > > is that it occurs more frequently on pages which utilize more than one > > connection. But thats about as far as I can narrow it. I've tried the > > 8.15 and 8.17 type4 jdbc drivers from oracle and we've tried Oracles > > ConnectionCacheImpl and Orions XADataSource implimentation, both show the > > same behavior (though both are using the Oracle Driver). I've also tried > > Orions jdbc debug and it shows nothing of interest. > > > > So far I've put about a week straight into finding it, and I've just about > > run out of ideas. I'd really be appreciative if anyone has any good > > suggestions on where to look. ANyone seen behavior like this before? > > > > > > >
RE: Performance problems (More Info).
JVM Settings -- My normal settings are "-server -Xincgc -Xms128m -Xmx384m". I've tried playing around with different JVM options including turning off server, using normal gc, running in interpreted mode, running in classic mode, et al. I even tried running under JDK1.2.2. While these definately effect performance in dramatic ways, none seems to fix this problem. Garbage Collection -- I tried watching the heap by turning on verbosegc, and the pauses do not appear to be timed with a full gc. So thats not it either, definately a good thought though... DB Server -- The DB server itself is performing fine, CPU and memory utilization is low. The database is completely accessable via sqlplus or an alternative app while these pauses occur. Connection Pooling vs Orions DataSource Manager -- Interestingly the problem seems to go away when I turn off Orions datasources, and run with a third party ConnectionPool manager (such as BitMechanic).While this fixes my immediate problem it will prevent me from using Orion as an EJB container, which is not good. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]]On Behalf Of Aaron > Tavistock > Sent: Monday, March 26, 2001 1:09 PM > To: Orion-Interest > Subject: Performance problems... > > > I've been working on getting Orion running in a production > environment for a > little while now and just when I thought everything was working > fine I go to > push to production and something load/volume related is creating massive > slowdowns. > > Basically every 250 database accesses or so there is a long pause > (20 to 60 > second), where nothing occurs. During this pause the CPU load *drops* to > practically nothing and our entire site is frozen. I'm not sure exactly > where the problem exists; it could be our code, our > configuration, or even a > bug in Orion. > > The environment is Redhat 6.2, JDK1.3, Oracle 8i. Its a pure > J2EE app, but > we're not using EJB. I initially thought it might be a memory issue, but > I've played with the JDK heap size and carefully watched memory > utilization > and thats also not the issue. I even considered that maybe > Evermind/IronFlare might have a throttle (to push you to get a > license) so I > put one of our production licenses on the QA box. > > I've since gotten a load tester and can reproduce the problem. Oddly, it > only happens on pages which require database access. Even more > interesting > is that it occurs more frequently on pages which utilize more than one > connection. But thats about as far as I can narrow it. I've tried the > 8.15 and 8.17 type4 jdbc drivers from oracle and we've tried Oracles > ConnectionCacheImpl and Orions XADataSource implimentation, both show the > same behavior (though both are using the Oracle Driver). I've also tried > Orions jdbc debug and it shows nothing of interest. > > So far I've put about a week straight into finding it, and I've just about > run out of ideas. I'd really be appreciative if anyone has any good > suggestions on where to look. ANyone seen behavior like this before? > >
Re: Performance problems...
A better solution is to figure out why you have so many objects. GC cycles increase linearly with the the more memory you allocate. If you allocate 2GB of memory, then it's normal to hav 10-20 second GC cycles. If you're getting values that are that high (with hotspot) then you need to profile your code and optimise. On Mon, 26 Mar 2001, Salvatore Sferrazza wrote: > I've had similar symptoms with ATG Dynamo. It usually occurs when the VM > decides to garbage collect. The way we get around this is to have > multiple Dynamo instances each with it's own dedicated VM and CPU. This > makes the user experience more acceptable across all sessions on the > system since the garbage collection per user is less noticeable. are you > only using 1 VM right now? > > just a thought. > > Sal > > >
Re: Performance problems...
I've had similar symptoms with ATG Dynamo. It usually occurs when the VM decides to garbage collect. The way we get around this is to have multiple Dynamo instances each with it's own dedicated VM and CPU. This makes the user experience more acceptable across all sessions on the system since the garbage collection per user is less noticeable. are you only using 1 VM right now? just a thought. Sal
RE: Performance problems...
And from totally out in left field, how about conflicting database transaction/table locks. I doubt it, because those tend to be indefinite, but it's a thought anyway! I suspect that the running-out-of- connections idea is more likely to be correct, though... Gary > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]]On Behalf Of Aaron > Tavistock > Sent: Sunday, March 25, 2001 7:09 PM > To: Orion-Interest > Subject: Performance problems... > > > I've been working on getting Orion running in a production environment for a > little while now and just when I thought everything was working fine I go to > push to production and something load/volume related is creating massive > slowdowns. > > Basically every 250 database accesses or so there is a long pause (20 to 60 > second), where nothing occurs. During this pause the CPU load *drops* to > practically nothing and our entire site is frozen. I'm not sure exactly > where the problem exists; it could be our code, our configuration, or even a > bug in Orion. > > The environment is Redhat 6.2, JDK1.3, Oracle 8i. Its a pure J2EE app, but > we're not using EJB. I initially thought it might be a memory issue, but > I've played with the JDK heap size and carefully watched memory utilization > and thats also not the issue. I even considered that maybe > Evermind/IronFlare might have a throttle (to push you to get a license) so I > put one of our production licenses on the QA box. > > I've since gotten a load tester and can reproduce the problem. Oddly, it > only happens on pages which require database access. Even more interesting > is that it occurs more frequently on pages which utilize more than one > connection. But thats about as far as I can narrow it. I've tried the > 8.15 and 8.17 type4 jdbc drivers from oracle and we've tried Oracles > ConnectionCacheImpl and Orions XADataSource implimentation, both show the > same behavior (though both are using the Oracle Driver). I've also tried > Orions jdbc debug and it shows nothing of interest. > > So far I've put about a week straight into finding it, and I've just about > run out of ideas. I'd really be appreciative if anyone has any good > suggestions on where to look. ANyone seen behavior like this before? > > >
RE: Performance problems...
What is the maximum number of connections that you have configured your database pool to? Could you be running out of connections? Are you properly releasing the connections back to the pool, and closing results set which might not be needed? During the time that the site is "frozen" (as you indicate 20 to 60 seconds) can you hit a page that is not database related (a simple .JSP perhaps)? Does that return fast (this would determine if your problem is database or Orion related). Have you tried to examine what's happening on the database server side when this occurs? Just some thoughts. -AP_ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Aaron Tavistock Sent: Sunday, March 25, 2001 7:09 PM To: Orion-Interest Subject: Performance problems... I've been working on getting Orion running in a production environment for a little while now and just when I thought everything was working fine I go to push to production and something load/volume related is creating massive slowdowns. Basically every 250 database accesses or so there is a long pause (20 to 60 second), where nothing occurs. During this pause the CPU load *drops* to practically nothing and our entire site is frozen. I'm not sure exactly where the problem exists; it could be our code, our configuration, or even a bug in Orion. The environment is Redhat 6.2, JDK1.3, Oracle 8i. Its a pure J2EE app, but we're not using EJB. I initially thought it might be a memory issue, but I've played with the JDK heap size and carefully watched memory utilization and thats also not the issue. I even considered that maybe Evermind/IronFlare might have a throttle (to push you to get a license) so I put one of our production licenses on the QA box. I've since gotten a load tester and can reproduce the problem. Oddly, it only happens on pages which require database access. Even more interesting is that it occurs more frequently on pages which utilize more than one connection. But thats about as far as I can narrow it. I've tried the 8.15 and 8.17 type4 jdbc drivers from oracle and we've tried Oracles ConnectionCacheImpl and Orions XADataSource implimentation, both show the same behavior (though both are using the Oracle Driver). I've also tried Orions jdbc debug and it shows nothing of interest. So far I've put about a week straight into finding it, and I've just about run out of ideas. I'd really be appreciative if anyone has any good suggestions on where to look. ANyone seen behavior like this before?
RE: Performance problems...
Q: Are you running the JVM with -server? Q: Could it be a garbage collection related problem? If it's happening at odd times when heap usage is up, it might be gc collecting old jdbc objects? -mike > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]]On Behalf Of Aaron > Tavistock > Sent: Monday, March 26, 2001 1:09 PM > To: Orion-Interest > Subject: Performance problems... > > > I've been working on getting Orion running in a production > environment for a > little while now and just when I thought everything was working > fine I go to > push to production and something load/volume related is creating massive > slowdowns. > > Basically every 250 database accesses or so there is a long pause > (20 to 60 > second), where nothing occurs. During this pause the CPU load *drops* to > practically nothing and our entire site is frozen. I'm not sure exactly > where the problem exists; it could be our code, our > configuration, or even a > bug in Orion. > > The environment is Redhat 6.2, JDK1.3, Oracle 8i. Its a pure > J2EE app, but > we're not using EJB. I initially thought it might be a memory issue, but > I've played with the JDK heap size and carefully watched memory > utilization > and thats also not the issue. I even considered that maybe > Evermind/IronFlare might have a throttle (to push you to get a > license) so I > put one of our production licenses on the QA box. > > I've since gotten a load tester and can reproduce the problem. Oddly, it > only happens on pages which require database access. Even more > interesting > is that it occurs more frequently on pages which utilize more than one > connection. But thats about as far as I can narrow it. I've tried the > 8.15 and 8.17 type4 jdbc drivers from oracle and we've tried Oracles > ConnectionCacheImpl and Orions XADataSource implimentation, both show the > same behavior (though both are using the Oracle Driver). I've also tried > Orions jdbc debug and it shows nothing of interest. > > So far I've put about a week straight into finding it, and I've just about > run out of ideas. I'd really be appreciative if anyone has any good > suggestions on where to look. ANyone seen behavior like this before? > >
Performance problems...
I've been working on getting Orion running in a production environment for a little while now and just when I thought everything was working fine I go to push to production and something load/volume related is creating massive slowdowns. Basically every 250 database accesses or so there is a long pause (20 to 60 second), where nothing occurs. During this pause the CPU load *drops* to practically nothing and our entire site is frozen. I'm not sure exactly where the problem exists; it could be our code, our configuration, or even a bug in Orion. The environment is Redhat 6.2, JDK1.3, Oracle 8i. Its a pure J2EE app, but we're not using EJB. I initially thought it might be a memory issue, but I've played with the JDK heap size and carefully watched memory utilization and thats also not the issue. I even considered that maybe Evermind/IronFlare might have a throttle (to push you to get a license) so I put one of our production licenses on the QA box. I've since gotten a load tester and can reproduce the problem. Oddly, it only happens on pages which require database access. Even more interesting is that it occurs more frequently on pages which utilize more than one connection. But thats about as far as I can narrow it. I've tried the 8.15 and 8.17 type4 jdbc drivers from oracle and we've tried Oracles ConnectionCacheImpl and Orions XADataSource implimentation, both show the same behavior (though both are using the Oracle Driver). I've also tried Orions jdbc debug and it shows nothing of interest. So far I've put about a week straight into finding it, and I've just about run out of ideas. I'd really be appreciative if anyone has any good suggestions on where to look. ANyone seen behavior like this before?
