Re: Performance degradation with Derby Embedded under Tomcat vs Jetty
Are you sure that under Tomcat the application is really using the Tomcat connection pool? If not that would certainly account for the differences. Even if Tomcat DBCP is configured, checking that the pool configurations are similar would seem like an important check. -Brett On Mon, Apr 13, 2015 at 4:56 PM, Øyvind Harboe oyvind.har...@zylin.com wrote: I know the Tomcat setup isn't using the clientdriver when it slows down, because when I first tried to switch to the ClientDriver it failed. After I copied the derbyclient.xxx.jar to tomcat/lib/, it worked. On Mon, Apr 13, 2015 at 9:51 AM, Dyre Tjeldvoll dyre.tjeldv...@oracle.com wrote: On 04/13/2015 12:27 AM, Øyvind Harboe wrote: I found another crucial clue: if I use ClientDriver instead of EmbeddedDriver and connect to the Derby database running under Jetty, I get identical performance with Tomcat and Jetty. This is very strong indication that there is something about the combination of EmbeddedDriver and Tomcat that is gumming up the works. Could it be that your Tomcat setup always uses the ClientDriver for some reason? I don't know much about setting up Databases with AppServers, but based on the number of questions on SO and other places, it isn't trivial... -- Regards, Dyre -- Øyvind Harboe - Can Zylin Consulting help on your project? http://www.zylin.com/
Re: Performance degradation with Derby Embedded under Tomcat vs Jetty
I don't think that's the problem: 1. The same amount of heap is configured 2. I've checked with VisualVM and it uses much less than the provided memory(ca. 250mbyte vs. 1024mbyte heap). On Mon, Apr 13, 2015 at 4:54 AM, Bryan Pendleton bpendleton.de...@gmail.com wrote: I've tried to figure out what the differences are between these two environments without luck. Perhaps the amount of memory available to Derby is dramatically different in the one case than in the other, and Derby is working very hard to accomplish its tasks with insufficient memory? bryan -- Øyvind Harboe - Can Zylin Consulting help on your project? http://www.zylin.com/
Re: Performance degradation with Derby Embedded under Tomcat vs Jetty
I know the Tomcat setup isn't using the clientdriver when it slows down, because when I first tried to switch to the ClientDriver it failed. After I copied the derbyclient.xxx.jar to tomcat/lib/, it worked. On Mon, Apr 13, 2015 at 9:51 AM, Dyre Tjeldvoll dyre.tjeldv...@oracle.com wrote: On 04/13/2015 12:27 AM, Øyvind Harboe wrote: I found another crucial clue: if I use ClientDriver instead of EmbeddedDriver and connect to the Derby database running under Jetty, I get identical performance with Tomcat and Jetty. This is very strong indication that there is something about the combination of EmbeddedDriver and Tomcat that is gumming up the works. Could it be that your Tomcat setup always uses the ClientDriver for some reason? I don't know much about setting up Databases with AppServers, but based on the number of questions on SO and other places, it isn't trivial... -- Regards, Dyre -- Øyvind Harboe - Can Zylin Consulting help on your project? http://www.zylin.com/
Re: Performance degradation with Derby Embedded under Tomcat vs Jetty
I'll be trying to do more logging. Do you have any handy pointers to documentation? To log SQL statements seems like the most promising: https://db.apache.org/derby/docs/10.11/ref/rrefproper43517.html I didn't find anything to log connection IDs (distinct/count). I never tried JMX mbeans w/VisualVM, so that should be interesting.
Re: Performance degradation with Derby Embedded under Tomcat vs Jetty
And, by testing with a network server I think you can access JMX mbeans with VisualVM/JConsole to check connection counts. -- Kristian 13. apr. 2015 13:12 skrev Kristian Waagan krist...@apache.org: Hi Øivind, Have you turned on SQL-logging/-tracing in Derby to check the activity? Possible things to check: o connection IDs (distinct/count) o expensive connection validation queries (not all of these are direct SQL, check pool docs/conf) o commits? o different queries? While one would expect the load / queries to be the same, one never knows... In theory one would think the client driver was more affected by a pool misconfiguration. Regards, -- Kristian 13. apr. 2015 11:56 skrev Øyvind Harboe oyvind.har...@zylin.com: I've made them identical near as I can. It isn't the same implementation of the connection pool. Is it only EmbeddedDriver that would be negatively affected by a mis-configured connection pool? Is there some logging in Derby that I could enable to verify that things are set up correctly, i.e. that new connections are not made all the time? On Mon, Apr 13, 2015 at 11:16 AM, Brett Wooldridge brett.wooldri...@gmail.com wrote: Are you sure that under Tomcat the application is really using the Tomcat connection pool? If not that would certainly account for the differences. Even if Tomcat DBCP is configured, checking that the pool configurations are similar would seem like an important check. -Brett On Mon, Apr 13, 2015 at 4:56 PM, Øyvind Harboe oyvind.har...@zylin.com wrote: I know the Tomcat setup isn't using the clientdriver when it slows down, because when I first tried to switch to the ClientDriver it failed. After I copied the derbyclient.xxx.jar to tomcat/lib/, it worked. On Mon, Apr 13, 2015 at 9:51 AM, Dyre Tjeldvoll dyre.tjeldv...@oracle.com wrote: On 04/13/2015 12:27 AM, Øyvind Harboe wrote: I found another crucial clue: if I use ClientDriver instead of EmbeddedDriver and connect to the Derby database running under Jetty, I get identical performance with Tomcat and Jetty. This is very strong indication that there is something about the combination of EmbeddedDriver and Tomcat that is gumming up the works. Could it be that your Tomcat setup always uses the ClientDriver for some reason? I don't know much about setting up Databases with AppServers, but based on the number of questions on SO and other places, it isn't trivial... -- Regards, Dyre -- Øyvind Harboe - Can Zylin Consulting help on your project? http://www.zylin.com/ -- Øyvind Harboe - Can Zylin Consulting help on your project? http://www.zylin.com/
Re: Performance degradation with Derby Embedded under Tomcat vs Jetty
Hi Øivind, Have you turned on SQL-logging/-tracing in Derby to check the activity? Possible things to check: o connection IDs (distinct/count) o expensive connection validation queries (not all of these are direct SQL, check pool docs/conf) o commits? o different queries? While one would expect the load / queries to be the same, one never knows... In theory one would think the client driver was more affected by a pool misconfiguration. Regards, -- Kristian 13. apr. 2015 11:56 skrev Øyvind Harboe oyvind.har...@zylin.com: I've made them identical near as I can. It isn't the same implementation of the connection pool. Is it only EmbeddedDriver that would be negatively affected by a mis-configured connection pool? Is there some logging in Derby that I could enable to verify that things are set up correctly, i.e. that new connections are not made all the time? On Mon, Apr 13, 2015 at 11:16 AM, Brett Wooldridge brett.wooldri...@gmail.com wrote: Are you sure that under Tomcat the application is really using the Tomcat connection pool? If not that would certainly account for the differences. Even if Tomcat DBCP is configured, checking that the pool configurations are similar would seem like an important check. -Brett On Mon, Apr 13, 2015 at 4:56 PM, Øyvind Harboe oyvind.har...@zylin.com wrote: I know the Tomcat setup isn't using the clientdriver when it slows down, because when I first tried to switch to the ClientDriver it failed. After I copied the derbyclient.xxx.jar to tomcat/lib/, it worked. On Mon, Apr 13, 2015 at 9:51 AM, Dyre Tjeldvoll dyre.tjeldv...@oracle.com wrote: On 04/13/2015 12:27 AM, Øyvind Harboe wrote: I found another crucial clue: if I use ClientDriver instead of EmbeddedDriver and connect to the Derby database running under Jetty, I get identical performance with Tomcat and Jetty. This is very strong indication that there is something about the combination of EmbeddedDriver and Tomcat that is gumming up the works. Could it be that your Tomcat setup always uses the ClientDriver for some reason? I don't know much about setting up Databases with AppServers, but based on the number of questions on SO and other places, it isn't trivial... -- Regards, Dyre -- Øyvind Harboe - Can Zylin Consulting help on your project? http://www.zylin.com/ -- Øyvind Harboe - Can Zylin Consulting help on your project? http://www.zylin.com/
Re: Performance degradation with Derby Embedded under Tomcat vs Jetty
Have you put the app under a profiler to see what components are taking the most time in each container? Seems like the best way to get some facts on the table. On Sun, 2015-04-12 at 23:50 +0200, Øyvind Harboe wrote: Hi, I'm having problems with performance degrading dramatically when I deploy our application to a Tomcat container. The application uses Apache Cayenne ORM on top of Derby. Does anyone have any pointers on how I can debug this to figure out what I'm doing wrong? Generating a report which includes ca. 100 queries and thousands of records takes less than one second on Jetty, but in the Tomcat container it takes more than 10 seconds, so a 10x slowdown or so. I can't believe that Jetty is 10x faster than Tomcat. I've tried to figure out what the differences are between these two environments without luck. Jetty and Tomcat are equal in terms of: - same VM options - debug vs. non-debug mode - tests run on same machine side-by-side - same database files(copied to another location on the harddisk) Some differences: - Jetty is using org.apache.cayenne.conn.PoolManager rather than Tomcat's connection pools. When I suspend Tomcat in the debugger, the typical stack trace is something about fetching pages: BaseContainerHandle(Observable).addObserver(Observer) line: 82 StoredPage(BasePage).preLatch(BaseContainerHandle) line: not available StoredPage(BasePage).setExclusive(BaseContainerHandle) line: not available RAFContainer4(BaseContainer).latchPage(BaseContainerHandle, BasePage, boolean) line: not available RAFContainer4(FileContainer).latchPage(BaseContainerHandle, BasePage, boolean) line: not available RAFContainer4(FileContainer).getUserPage(BaseContainerHandle, long, boolean, boolean) line: not available RAFContainer4(FileContainer).getPage(BaseContainerHandle, long, boolean) line: not available BaseContainerHandle.getPage(long) line: not available OpenHeap(OpenConglomerate).latchPage(RowPosition) line: not available HeapController(GenericConglomerateController).fetch(RowLocation, DataValueDescriptor[], FormatableBitSet) line: not available IndexRowToBaseRowResultSet.getNextRowCore() line: not available ProjectRestrictResultSet.getNextRowCore() line: not available NestedLoopJoinResultSet.getNextRowCore() line: not available ProjectRestrictResultSet.getNextRowCore() line: not available SortResultSet.getRowFromResultSet() line: not available SortResultSet.getNextRowFromRS() line: not available SortResultSet.loadSorter() line: not available SortResultSet.openCore() line: not available SortResultSet(BasicNoPutResultSetImpl).open() line: not available GenericPreparedStatement.executeStmt(Activation, boolean, boolean, long) line: not available GenericPreparedStatement.execute(Activation, boolean, long) line: not available EmbedPreparedStatement(EmbedStatement).executeStatement(Activation, boolean, boolean) line: not available EmbedPreparedStatement.executeStatement(Activation, boolean, boolean) line: not available EmbedPreparedStatement.executeQuery() line: not available SelectAction.performAction(Connection, OperationObserver) line: 75 DataNodeQueryAction.runQuery(Connection, Query) line: 87
Re: Performance degradation with Derby Embedded under Tomcat vs Jetty
I did. See the stacktrace in my initial post. On Apr 13, 2015 3:18 PM, Tim Watts t...@cliftonfarm.org wrote: Have you put the app under a profiler to see what components are taking the most time in each container? Seems like the best way to get some facts on the table. On Sun, 2015-04-12 at 23:50 +0200, Øyvind Harboe wrote: Hi, I'm having problems with performance degrading dramatically when I deploy our application to a Tomcat container. The application uses Apache Cayenne ORM on top of Derby. Does anyone have any pointers on how I can debug this to figure out what I'm doing wrong? Generating a report which includes ca. 100 queries and thousands of records takes less than one second on Jetty, but in the Tomcat container it takes more than 10 seconds, so a 10x slowdown or so. I can't believe that Jetty is 10x faster than Tomcat. I've tried to figure out what the differences are between these two environments without luck. Jetty and Tomcat are equal in terms of: - same VM options - debug vs. non-debug mode - tests run on same machine side-by-side - same database files(copied to another location on the harddisk) Some differences: - Jetty is using org.apache.cayenne.conn.PoolManager rather than Tomcat's connection pools. When I suspend Tomcat in the debugger, the typical stack trace is something about fetching pages: BaseContainerHandle(Observable).addObserver(Observer) line: 82 StoredPage(BasePage).preLatch(BaseContainerHandle) line: not available StoredPage(BasePage).setExclusive(BaseContainerHandle) line: not available RAFContainer4(BaseContainer).latchPage(BaseContainerHandle, BasePage, boolean) line: not available RAFContainer4(FileContainer).latchPage(BaseContainerHandle, BasePage, boolean) line: not available RAFContainer4(FileContainer).getUserPage(BaseContainerHandle, long, boolean, boolean) line: not available RAFContainer4(FileContainer).getPage(BaseContainerHandle, long, boolean) line: not available BaseContainerHandle.getPage(long) line: not available OpenHeap(OpenConglomerate).latchPage(RowPosition) line: not available HeapController(GenericConglomerateController).fetch(RowLocation, DataValueDescriptor[], FormatableBitSet) line: not available IndexRowToBaseRowResultSet.getNextRowCore() line: not available ProjectRestrictResultSet.getNextRowCore() line: not available NestedLoopJoinResultSet.getNextRowCore() line: not available ProjectRestrictResultSet.getNextRowCore() line: not available SortResultSet.getRowFromResultSet() line: not available SortResultSet.getNextRowFromRS() line: not available SortResultSet.loadSorter() line: not available SortResultSet.openCore() line: not available SortResultSet(BasicNoPutResultSetImpl).open() line: not available GenericPreparedStatement.executeStmt(Activation, boolean, boolean, long) line: not available GenericPreparedStatement.execute(Activation, boolean, long) line: not available EmbedPreparedStatement(EmbedStatement).executeStatement(Activation, boolean, boolean) line: not available EmbedPreparedStatement.executeStatement(Activation, boolean, boolean) line: not available EmbedPreparedStatement.executeQuery() line: not available SelectAction.performAction(Connection, OperationObserver) line: 75 DataNodeQueryAction.runQuery(Connection, Query) line: 87
