Re: performance slowdown for 100,000 users, all with same PSML
I don't mind at all :) On Tue, 16 Nov 2004 16:00:00 -0800, David Sean Taylor [EMAIL PROTECTED] wrote: Youssef Mohammed wrote: I think it has nothing to do with the portlets since the only changes he (Dan) made was the number of users. I suggest to do some profiling (both IBM and Oracle stuff can help ) to the jetspped instance to see what is going on. I used to work on J1 last year and I did found some scalability issues on it. I donno if they still exist on not. If our resident performance expert doesn't mind, could I also recommend trying the delay rendering feature, which allows for J1 to render portlets in parallel. By default, J1 will render portlet sequentially, meaning that portlet 2 doesnt start rendering until portet 1 completes and so on. -- David Sean Taylor Bluesunrise Software [EMAIL PROTECTED] [office] +01 707 773 4646 [mobile] +01 707 529 9194 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Regards, Youssef - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: performance slowdown for 100,000 users, all with same PSML
Well folks, we found the issue with the 100,000 users problem. I used p6spy to grab all the SQL statements going to the database, and we saw this: 1099788107943|13|1|statement||SELECT TURBINE_USER.USER_ID, TURBINE_USER.LOGIN_NA ME, TURBINE_USER.PASSWORD_VALUE, TURBINE_USER.FIRST_NAME, TURBINE_USER.LAST_NAME , TURBINE_USER.EMAIL, TURBINE_USER.CONFIRM_VALUE, TURBINE_USER.MODIFIED, TURBINE _USER.CREATED, TURBINE_USER.LAST_LOGIN, TURBINE_USER.DISABLED, TURBINE_USER.OBJE CTDATA, TURBINE_USER.PASSWORD_CHANGED FROM TURBINE_USER WHERE TURBINE_USER.LOGIN _NAME='17215' 1099788107976|15|1|statement||SELECT TURBINE_USER_GROUP_ROLE.USER_ID, TURBINE_US ER_GROUP_ROLE.GROUP_ID, TURBINE_USER_GROUP_ROLE.ROLE_ID FROM TURBINE_USER_GROUP_ ROLE WHERE TURBINE_USER_GROUP_ROLE.USER_ID='17215' 1099788108002|10|1|statement||SELECT TURBINE_ROLE.ROLE_ID, TURBINE_ROLE.ROLE_NAM E, TURBINE_ROLE.OBJECTDATA FROM TURBINE_ROLE WHERE TURBINE_ROLE.ROLE_ID=1 1099788108032|10|1|statement||SELECT TURBINE_GROUP.GROUP_ID, TURBINE_GROUP.GROUP _NAME, TURBINE_GROUP.OBJECTDATA FROM TURBINE_GROUP WHERE TURBINE_GROUP.GROUP_ID=1 Many many times. In fact, these four statements were repeated around 60 times for one page load with 9 portlets. I don't know why, since this property was set in JetspeedSecurity.properties: services.JetspeedSecurity.caching.enable=true This obviously slowed portlet rendering down quite a bit. Since we are not using any of Jetspeed's authorization capabilities, we turned it off by setting this property: services.PortalAccessController.classname=org.apache.jetspeed.services.security. nosecurity.NoSecurityAccessController This helped a lot. We saw no more of the statements mentioned above, and page load times go down dramatically. Thanks for all your help. Dan --- David Sean Taylor [EMAIL PROTECTED] wrote: Youssef Mohammed wrote: I think it has nothing to do with the portlets since the only changes he (Dan) made was the number of users. I suggest to do some profiling (both IBM and Oracle stuff can help ) to the jetspped instance to see what is going on. I used to work on J1 last year and I did found some scalability issues on it. I donno if they still exist on not. If our resident performance expert doesn't mind, could I also recommend trying the delay rendering feature, which allows for J1 to render portlets in parallel. By default, J1 will render portlet sequentially, meaning that portlet 2 doesnt start rendering until portet 1 completes and so on. -- David Sean Taylor Bluesunrise Software [EMAIL PROTECTED] [office] +01 707 773 4646 [mobile] +01 707 529 9194 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: performance slowdown for 100,000 users, all with same PSML
Glad that u solved your problem. I also wonder if anyone has explanation of what happened to Dan and other solution without turning the SecuirtyAccessControl off. On Wed, 17 Nov 2004 08:58:46 -0800 (PST), Dan Moore [EMAIL PROTECTED] wrote: Well folks, we found the issue with the 100,000 users problem. I used p6spy to grab all the SQL statements going to the database, and we saw this: 1099788107943|13|1|statement||SELECT TURBINE_USER.USER_ID, TURBINE_USER.LOGIN_NA ME, TURBINE_USER.PASSWORD_VALUE, TURBINE_USER.FIRST_NAME, TURBINE_USER.LAST_NAME , TURBINE_USER.EMAIL, TURBINE_USER.CONFIRM_VALUE, TURBINE_USER.MODIFIED, TURBINE _USER.CREATED, TURBINE_USER.LAST_LOGIN, TURBINE_USER.DISABLED, TURBINE_USER.OBJE CTDATA, TURBINE_USER.PASSWORD_CHANGED FROM TURBINE_USER WHERE TURBINE_USER.LOGIN _NAME='17215' 1099788107976|15|1|statement||SELECT TURBINE_USER_GROUP_ROLE.USER_ID, TURBINE_US ER_GROUP_ROLE.GROUP_ID, TURBINE_USER_GROUP_ROLE.ROLE_ID FROM TURBINE_USER_GROUP_ ROLE WHERE TURBINE_USER_GROUP_ROLE.USER_ID='17215' 1099788108002|10|1|statement||SELECT TURBINE_ROLE.ROLE_ID, TURBINE_ROLE.ROLE_NAM E, TURBINE_ROLE.OBJECTDATA FROM TURBINE_ROLE WHERE TURBINE_ROLE.ROLE_ID=1 1099788108032|10|1|statement||SELECT TURBINE_GROUP.GROUP_ID, TURBINE_GROUP.GROUP _NAME, TURBINE_GROUP.OBJECTDATA FROM TURBINE_GROUP WHERE TURBINE_GROUP.GROUP_ID=1 Many many times. In fact, these four statements were repeated around 60 times for one page load with 9 portlets. I don't know why, since this property was set in JetspeedSecurity.properties: services.JetspeedSecurity.caching.enable=true This obviously slowed portlet rendering down quite a bit. Since we are not using any of Jetspeed's authorization capabilities, we turned it off by setting this property: services.PortalAccessController.classname=org.apache.jetspeed.services.security. nosecurity.NoSecurityAccessController This helped a lot. We saw no more of the statements mentioned above, and page load times go down dramatically. Thanks for all your help. Dan --- David Sean Taylor [EMAIL PROTECTED] wrote: Youssef Mohammed wrote: I think it has nothing to do with the portlets since the only changes he (Dan) made was the number of users. I suggest to do some profiling (both IBM and Oracle stuff can help ) to the jetspped instance to see what is going on. I used to work on J1 last year and I did found some scalability issues on it. I donno if they still exist on not. If our resident performance expert doesn't mind, could I also recommend trying the delay rendering feature, which allows for J1 to render portlets in parallel. By default, J1 will render portlet sequentially, meaning that portlet 2 doesnt start rendering until portet 1 completes and so on. -- David Sean Taylor Bluesunrise Software [EMAIL PROTECTED] [office] +01 707 773 4646 [mobile] +01 707 529 9194 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Regards, Youssef - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: performance slowdown for 100,000 users, all with same PSML
I can verify that I saw this behavior on PostgreSQL back on 1.4, so I suspect it's not DB dependant. I use a custom authentication class and do my own caching, so never really pursued it. (And, in fact, forgot about it until this message jogged my memory.) -- Michael On 11/17/04 8:58 AM, Dan Moore [EMAIL PROTECTED] wrote: Well folks, we found the issue with the 100,000 users problem. I used p6spy to grab all the SQL statements going to the database, and we saw this: 1099788107943|13|1|statement||SELECT TURBINE_USER.USER_ID, TURBINE_USER.LOGIN_NA ME, TURBINE_USER.PASSWORD_VALUE, TURBINE_USER.FIRST_NAME, TURBINE_USER.LAST_NAME , TURBINE_USER.EMAIL, TURBINE_USER.CONFIRM_VALUE, TURBINE_USER.MODIFIED, TURBINE _USER.CREATED, TURBINE_USER.LAST_LOGIN, TURBINE_USER.DISABLED, TURBINE_USER.OBJE CTDATA, TURBINE_USER.PASSWORD_CHANGED FROM TURBINE_USER WHERE TURBINE_USER.LOGIN _NAME='17215' 1099788107976|15|1|statement||SELECT TURBINE_USER_GROUP_ROLE.USER_ID, TURBINE_US ER_GROUP_ROLE.GROUP_ID, TURBINE_USER_GROUP_ROLE.ROLE_ID FROM TURBINE_USER_GROUP_ ROLE WHERE TURBINE_USER_GROUP_ROLE.USER_ID='17215' 1099788108002|10|1|statement||SELECT TURBINE_ROLE.ROLE_ID, TURBINE_ROLE.ROLE_NAM E, TURBINE_ROLE.OBJECTDATA FROM TURBINE_ROLE WHERE TURBINE_ROLE.ROLE_ID=1 1099788108032|10|1|statement||SELECT TURBINE_GROUP.GROUP_ID, TURBINE_GROUP.GROUP _NAME, TURBINE_GROUP.OBJECTDATA FROM TURBINE_GROUP WHERE TURBINE_GROUP.GROUP_ID=1 Many many times. In fact, these four statements were repeated around 60 times for one page load with 9 portlets. I don't know why, since this property was set in JetspeedSecurity.properties: services.JetspeedSecurity.caching.enable=true This obviously slowed portlet rendering down quite a bit. Since we are not using any of Jetspeed's authorization capabilities, we turned it off by setting this property: services.PortalAccessController.classname=org.apache.jetspeed.services.securit y. nosecurity.NoSecurityAccessController This helped a lot. We saw no more of the statements mentioned above, and page load times go down dramatically. Thanks for all your help. Dan --- David Sean Taylor [EMAIL PROTECTED] wrote: Youssef Mohammed wrote: I think it has nothing to do with the portlets since the only changes he (Dan) made was the number of users. I suggest to do some profiling (both IBM and Oracle stuff can help ) to the jetspped instance to see what is going on. I used to work on J1 last year and I did found some scalability issues on it. I donno if they still exist on not. If our resident performance expert doesn't mind, could I also recommend trying the delay rendering feature, which allows for J1 to render portlets in parallel. By default, J1 will render portlet sequentially, meaning that portlet 2 doesnt start rendering until portet 1 completes and so on. -- David Sean Taylor Bluesunrise Software [EMAIL PROTECTED] [office] +01 707 773 4646 [mobile] +01 707 529 9194 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: performance slowdown for 100,000 users, all with same PSML
Youssef Mohammed wrote: I think it has nothing to do with the portlets since the only changes he (Dan) made was the number of users. I suggest to do some profiling (both IBM and Oracle stuff can help ) to the jetspped instance to see what is going on. I used to work on J1 last year and I did found some scalability issues on it. I donno if they still exist on not. If our resident performance expert doesn't mind, could I also recommend trying the delay rendering feature, which allows for J1 to render portlets in parallel. By default, J1 will render portlet sequentially, meaning that portlet 2 doesnt start rendering until portet 1 completes and so on. -- David Sean Taylor Bluesunrise Software [EMAIL PROTECTED] [office] +01 707 773 4646 [mobile] +01 707 529 9194 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: performance slowdown for 100,000 users, all with same PSML
David Sean Taylor wrote: Youssef Mohammed wrote: I think it has nothing to do with the portlets since the only changes he (Dan) made was the number of users. I suggest to do some profiling (both IBM and Oracle stuff can help ) to the jetspped instance to see what is going on. I used to work on J1 last year and I did found some scalability issues on it. I donno if they still exist on not. If our resident performance expert doesn't mind, could I also recommend trying the delay rendering feature, which allows for J1 to render portlets in parallel. By default, J1 will render portlet sequentially, meaning that portlet 2 doesnt start rendering until portet 1 completes and so on. The delayed-rendering feature works the best in the situation when you have many portlets on your page and some of which are much slower than others. Then you can set those slow ones to be delay-rendered. The effect is your page appear when all the fast portlets are finished rendering and then the slow ones will be requested from the server and rendered. Users get better perceived performance. Notice this feature actually increases the total number of requests going into the server. Jun - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: performance slowdown for 100,000 users, all with same PSML
Hi: I don't know about J1 scalability issues, but HSQL does. Could the DB (in a slow computer and a JVM not optimized for memory) be the problem in this case? HyperSonic (HSQLDB): (...) Hypersonic was originally designed as an in-memory database; as a result, operations performed on tables that are larger than available memory tend to run very slowly. This scalability issue relegates Hypersonic to the requirements of small applications. http://www.devx.com/IBMCloudscape/Article/21773/4165?pf=true Hypersonic works a client at a time: With other database engines, connection pools are used for reasons that may not apply to HSQLDB. * To allow new queries to be performed while a time-consuming query is being performed in the background. This is not possible with HSQLDB as it blocks while performing the first query and deals with the next query once it has finished it. http://hsqldb.sourceforge.net/doc/guide/ch04.html -Original Message- From: Dan Moore [mailto:[EMAIL PROTECTED] Sent: Monday, November 15, 2004 7:04 PM To: [EMAIL PROTECTED] Subject: performance slowdown for 100,000 users, all with same PSML Hi folks, I have an application I'm building on top of Jetspeed 1.5--I just grabbed the WAR from portals.apache.org. The back end is an Oracle database. I end up having around 100,000 users in the turbine_user table. All of these users have the same PSML and they all have the same role; we're using the role fallback mechanism. The PSML is stored on the filesystem, under role/user/language/country; right now I'm only using English and the US. I was seeing a massive slowdown in our application when we went from the 3 sample users to the full 100,000, so I'm trouble shooting that. The average page draw went from 4 seconds to 18 seconds (this is with 4 simulated users hitting the home page at once). I was able to get response times of around 4 seconds on my box (it's old and slow) with the default jetspeed setup (just as downloaded from the site: hypersonic db, only logging in as an anonymous user). I modified the JetspeedSessionValidator to log folks in with a cookie, but that's the only code change I've made to the vanilla jetspeed war I'm benchmarking (I profiled it and that method only adds 300 milliseconds or so). I did make two configuration changes: automatic.logon.enable=true database.default.maxConnections=20 I took a look at this page: http://nagoya.apache.org/wiki/apachewiki.cgi?JetspeedPerformanceTips but didn't really notice any huge gains (I did up the number of connections allowed in TurbineResources.properties and Torque.properties). In addition, it seems that many of the suggestions have been incorporated into the defaults. The turbine_user and turbine_user_group_role tables have their standard indices. I'm trying to figure out where the extra 15 seconds comes from. I guess I'm not so concerned about the absolute time it takes (we're going to deploy on much faster hardware), but I do want to get the ratio down. From just watching the page draw, it seems that the top gets drawn quickly, then the actual portlets themselves are slow. I see warnings in the logs about the specific psml files not being found for a given user--are these a significant slowdown? Should I change services.Profiler.fallback.country and services.Profiler.fallback.language to be false? For one PSML file, is db based PSML a better choice? Is there some caching that I can turn on that isn't on by default? Is simpler PSML going to cause any kind of speed up? I also searched the mailing list for performance related issues. I saw this (http://nagoya.apache.org/eyebrowse/ReadMsg?listName=jetspeed-user @jakarta.apache.orgmsgId=1640251) which I plan to try tomorrow morning, but didn't see any other things I should be doing. Any help would be most appreciated. Thanks, Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: performance slowdown for 100,000 users, all with same PSML
Hi Dan, Try having zero portlets on the default page, and see if it makes a difference... Then start adding your portlets in and see where the bottleneck starts. Dan Moore wrote: Hi folks, I have an application I'm building on top of Jetspeed 1.5--I just grabbed the WAR from portals.apache.org. The back end is an Oracle database. I end up having around 100,000 users in the turbine_user table. All of these users have the same PSML and they all have the same role; we're using the role fallback mechanism. The PSML is stored on the filesystem, under role/user/language/country; right now I'm only using English and the US. I was seeing a massive slowdown in our application when we went from the 3 sample users to the full 100,000, so I'm trouble shooting that. The average page draw went from 4 seconds to 18 seconds (this is with 4 simulated users hitting the home page at once). I was able to get response times of around 4 seconds on my box (it's old and slow) with the default jetspeed setup (just as downloaded from the site: hypersonic db, only logging in as an anonymous user). I modified the JetspeedSessionValidator to log folks in with a cookie, but that's the only code change I've made to the vanilla jetspeed war I'm benchmarking (I profiled it and that method only adds 300 milliseconds or so). I did make two configuration changes: automatic.logon.enable=true database.default.maxConnections=20 I took a look at this page: http://nagoya.apache.org/wiki/apachewiki.cgi?JetspeedPerformanceTips but didn't really notice any huge gains (I did up the number of connections allowed in TurbineResources.properties and Torque.properties). In addition, it seems that many of the suggestions have been incorporated into the defaults. The turbine_user and turbine_user_group_role tables have their standard indices. I'm trying to figure out where the extra 15 seconds comes from. I guess I'm not so concerned about the absolute time it takes (we're going to deploy on much faster hardware), but I do want to get the ratio down. From just watching the page draw, it seems that the top gets drawn quickly, then the actual portlets themselves are slow. I see warnings in the logs about the specific psml files not being found for a given user--are these a significant slowdown? Should I change services.Profiler.fallback.country and services.Profiler.fallback.language to be false? For one PSML file, is db based PSML a better choice? Is there some caching that I can turn on that isn't on by default? Is simpler PSML going to cause any kind of speed up? I also searched the mailing list for performance related issues. I saw this (http://nagoya.apache.org/eyebrowse/[EMAIL PROTECTED]msgId=1640251) which I plan to try tomorrow morning, but didn't see any other things I should be doing. Any help would be most appreciated. Thanks, Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- David Sean Taylor Bluesunrise Software [EMAIL PROTECTED] [office] +01 707 773 4646 [mobile] +01 707 529 9194 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: performance slowdown for 100,000 users, all with same PSML
I think it has nothing to do with the portlets since the only changes he (Dan) made was the number of users. I suggest to do some profiling (both IBM and Oracle stuff can help ) to the jetspped instance to see what is going on. I used to work on J1 last year and I did found some scalability issues on it. I donno if they still exist on not. Regards , On Mon, 15 Nov 2004 18:15:49 -0800, David Sean Taylor [EMAIL PROTECTED] wrote: Hi Dan, Try having zero portlets on the default page, and see if it makes a difference... Then start adding your portlets in and see where the bottleneck starts. Dan Moore wrote: Hi folks, I have an application I'm building on top of Jetspeed 1.5--I just grabbed the WAR from portals.apache.org. The back end is an Oracle database. I end up having around 100,000 users in the turbine_user table. All of these users have the same PSML and they all have the same role; we're using the role fallback mechanism. The PSML is stored on the filesystem, under role/user/language/country; right now I'm only using English and the US. I was seeing a massive slowdown in our application when we went from the 3 sample users to the full 100,000, so I'm trouble shooting that. The average page draw went from 4 seconds to 18 seconds (this is with 4 simulated users hitting the home page at once). I was able to get response times of around 4 seconds on my box (it's old and slow) with the default jetspeed setup (just as downloaded from the site: hypersonic db, only logging in as an anonymous user). I modified the JetspeedSessionValidator to log folks in with a cookie, but that's the only code change I've made to the vanilla jetspeed war I'm benchmarking (I profiled it and that method only adds 300 milliseconds or so). I did make two configuration changes: automatic.logon.enable=true database.default.maxConnections=20 I took a look at this page: http://nagoya.apache.org/wiki/apachewiki.cgi?JetspeedPerformanceTips but didn't really notice any huge gains (I did up the number of connections allowed in TurbineResources.properties and Torque.properties). In addition, it seems that many of the suggestions have been incorporated into the defaults. The turbine_user and turbine_user_group_role tables have their standard indices. I'm trying to figure out where the extra 15 seconds comes from. I guess I'm not so concerned about the absolute time it takes (we're going to deploy on much faster hardware), but I do want to get the ratio down. From just watching the page draw, it seems that the top gets drawn quickly, then the actual portlets themselves are slow. I see warnings in the logs about the specific psml files not being found for a given user--are these a significant slowdown? Should I change services.Profiler.fallback.country and services.Profiler.fallback.language to be false? For one PSML file, is db based PSML a better choice? Is there some caching that I can turn on that isn't on by default? Is simpler PSML going to cause any kind of speed up? I also searched the mailing list for performance related issues. I saw this (http://nagoya.apache.org/eyebrowse/[EMAIL PROTECTED]msgId=1640251) which I plan to try tomorrow morning, but didn't see any other things I should be doing. Any help would be most appreciated. Thanks, Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- David Sean Taylor Bluesunrise Software [EMAIL PROTECTED] [office] +01 707 773 4646 [mobile] +01 707 529 9194 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Regards, Youssef - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
