Re: mod_ajp and Load-Balancing Issue
On Mon, Sep 22, 2008 at 1:45 AM, Shaun Senecal [EMAIL PROTECTED] wrote: Are you using the ClusterSingleSignOn Valve? I'm not, no. I have httpd 2.2 on the front-end, with a balancer pool thus: ProxyRequests Off ProxyVia Off ProxyPreserveHost On ProxyErrorOverride On ProxyTimeout 60 Proxy balancer://tomcat BalancerMember ajp://10.1.1.231:8009 min=10 max=50 BalancerMember ajp://10.1.1.232:8009 min=10 max=50 #BalancerMember ajp://10.1.1.233:8009 min=10 max=50 Order deny,allow Allow from all /Proxy ProxyPass / balancer://tomcat/ lbmethod=byrequests stickysession=JSESSIONID nofailover=On ProxyPass /servlet balancer://tomcat/servlet lbmethod=byrequests stickysession=JSESSIONID nofailover=On ProxyPass /piston balancer://tomcat/piston lbmethod=byrequests stickysession=JSESSIONID nofailover=On ProxyPass /manager balancer://tomcat/manager lbmethod=byrequests stickysession=JSESSIONID nofailover=On ProxyPass /link balancer://tomcat/link lbmethod=byrequests stickysession=JSESSIONID nofailover=On ProxyPass /ajax balancer://tomcat/ajax lbmethod=byrequests stickysession=JSESSIONID nofailover=On When I browse to the login page, I see a jsessionid in firebug. I login, the jessionid cookie stays the same. I try to navigate within the application, and wherever I click I end up on the login screen again, with the message user has timed out. The cookie doesn't change until I login, in which case I see a new cookie. If I comment out two of the BalancerMembers, the app works fine. If I add one in (as above) I get the same behaviour. From reading the archives, it seems I should set a route in httpd.conf snippet and a jvmroute in the engine tag for each tomcat server.xml. I'll try this shortly, but it seems as if Richard did this, but still experienced problems. Thanks, S. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_ajp and Load-Balancing Issue
Good morning, By error, I just meant that I'd get redirected to the login page instead of the expected page. Sorry to confuse. There are no error pages, logs or messages. Just that I got switched to a different node. snip If you want to debug a little more: In Tomcat you can add a %S to your log pattern, which will log the session id. In httpd you can log the Set-Cookie outgoing header %{Set-Cookie}o and the JSESSIONID cookie %{JSESSIONID}C. If you are not using cookies, you can of course see the jsessionid path parameter dircetly in the logged URL. Thanks. I'll do that. (First time apache troubleshooter here). I'll get back on the results. Did you get this to work? I have exactly the same problem. Apache 2.2 using mod_proxy_ajp with 3 x tomcat 6 instances behind it. If I remove two of the tomcats from the balancer pool, the application works. When I put them back in, I can log into one of the servers, but as soon as I change screens on the application, I'm logged out. This is because if the application detects a session change, it logs the user out. I need the load-balancer to direct traffic to the same tomcat server on which the session began unless that server is down. S. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_ajp and Load-Balancing Issue
Are you using the ClusterSingleSignOn Valve? If you are, this sounds like the behaviour I was seeing, and have since resolved. The problem I had (well, part of the problem) was that the SSO information was not being replicated across the cluster when tomcat instances were brought back up. This meant that as long as the user was connecting to one of the -original- cluster instances everything was ok. However, as soon as the client gets directed to one of the newly brought up instances they have no SSO info and are prompted for login. The solution was relatively simple. I had to extend the ClusterSingleSignOn and ClusterSingleSignOnListener classes to ensure that A, when an instance is brought down the SSO information is not deactivated across the cluster and, B, that when an instance is brought back up that it syncs with the cluster to gather all currently known SSO info. I am planning on merging the information into the ClusterSingleSignOn and ClusterSingleSignOnListener classes and proposing a patch to Tomcat, but no one has responded to my original post and I havent had the chance to truley verify my fix. It seems to be running and has been for a while now, but I wouldnt put it into a production system just yet. S. On Sun, Sep 21, 2008 at 3:08 PM, Stephen Nelson-Smith [EMAIL PROTECTED]wrote: Good morning, By error, I just meant that I'd get redirected to the login page instead of the expected page. Sorry to confuse. There are no error pages, logs or messages. Just that I got switched to a different node. snip If you want to debug a little more: In Tomcat you can add a %S to your log pattern, which will log the session id. In httpd you can log the Set-Cookie outgoing header %{Set-Cookie}o and the JSESSIONID cookie %{JSESSIONID}C. If you are not using cookies, you can of course see the jsessionid path parameter dircetly in the logged URL. Thanks. I'll do that. (First time apache troubleshooter here). I'll get back on the results. Did you get this to work? I have exactly the same problem. Apache 2.2 using mod_proxy_ajp with 3 x tomcat 6 instances behind it. If I remove two of the tomcats from the balancer pool, the application works. When I put them back in, I can log into one of the servers, but as soon as I change screens on the application, I'm logged out. This is because if the application detects a session change, it logs the user out. I need the load-balancer to direct traffic to the same tomcat server on which the session began unless that server is down. S. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_ajp and Load-Balancing Issue
Oops. My orignal post is here ( http://www.nabble.com/Clustered-SSO-improperly-invalidated-upon-web-application-shutdown-to19447895.html#a19447895). It might descirbe the problem better to see if this is the same issue you are facing. On Mon, Sep 22, 2008 at 9:45 AM, Shaun Senecal [EMAIL PROTECTED]wrote: Are you using the ClusterSingleSignOn Valve? If you are, this sounds like the behaviour I was seeing, and have since resolved. The problem I had (well, part of the problem) was that the SSO information was not being replicated across the cluster when tomcat instances were brought back up. This meant that as long as the user was connecting to one of the -original- cluster instances everything was ok. However, as soon as the client gets directed to one of the newly brought up instances they have no SSO info and are prompted for login. The solution was relatively simple. I had to extend the ClusterSingleSignOn and ClusterSingleSignOnListener classes to ensure that A, when an instance is brought down the SSO information is not deactivated across the cluster and, B, that when an instance is brought back up that it syncs with the cluster to gather all currently known SSO info. I am planning on merging the information into the ClusterSingleSignOn and ClusterSingleSignOnListener classes and proposing a patch to Tomcat, but no one has responded to my original post and I havent had the chance to truley verify my fix. It seems to be running and has been for a while now, but I wouldnt put it into a production system just yet. S. On Sun, Sep 21, 2008 at 3:08 PM, Stephen Nelson-Smith [EMAIL PROTECTED]wrote: Good morning, By error, I just meant that I'd get redirected to the login page instead of the expected page. Sorry to confuse. There are no error pages, logs or messages. Just that I got switched to a different node. snip If you want to debug a little more: In Tomcat you can add a %S to your log pattern, which will log the session id. In httpd you can log the Set-Cookie outgoing header %{Set-Cookie}o and the JSESSIONID cookie %{JSESSIONID}C. If you are not using cookies, you can of course see the jsessionid path parameter dircetly in the logged URL. Thanks. I'll do that. (First time apache troubleshooter here). I'll get back on the results. Did you get this to work? I have exactly the same problem. Apache 2.2 using mod_proxy_ajp with 3 x tomcat 6 instances behind it. If I remove two of the tomcats from the balancer pool, the application works. When I put them back in, I can log into one of the servers, but as soon as I change screens on the application, I'm logged out. This is because if the application detects a session change, it logs the user out. I need the load-balancer to direct traffic to the same tomcat server on which the session began unless that server is down. S. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: mod_ajp and Load-Balancing Issue
I've been observing similar behavior. I think there is actually a bug in either apache mod_proxy or tomcat. Here is what happens: Sometimes and I'm not quite sure what condition triggers this in apache_proxy, apache_proxy modifies the cookie value by adding the name of the worker server so a cookie would look like this: {original.client.sent.cookie.value}.{worker_name} So mod_proxy suffixes the cookie value with .{worker_name}. Now then this cookie is sent to the correct server. When this cookie arrives at the server, server no longer can find a session stored against this moded cookie value and rejects the request. Then what most likely happens client gets an error back from the server (via proxy) and resets. Try to install an HTTP traffic sniffer (there are plugins for firefox) and monitor your cockie value on the client and you will see what I'm talking about. Marat Plana, Richard wrote: Basically, I open a user session and go through various pages on my app. After a couple of clicks, the app just brings me back to the start page. I'm monitoring logs from both sides and I see that as soon as it comes back in error and brings me to the login page, activity switches to the other app server (the one that I wasn't using prior to the problem). Just to make sure: what kind of error do you mean here? The httpd timeout and that is not the only source of lost stickyness, or some application error? If it is an application error, it would not be too strange, that the application invalidates the session. By error, I just meant that I'd get redirected to the login page instead of the expected page. Sorry to confuse. There are no error pages, logs or messages. Just that I got switched to a different node. Good catch, though. I've added nofailover=On to the ProxyPass line and tested it. Again, in the middle of the session, it switched to another node and I lost my session again. So not likely an error-induced failover. If you want to debug a little more: In Tomcat you can add a %S to your log pattern, which will log the session id. In httpd you can log the Set-Cookie outgoing header %{Set-Cookie}o and the JSESSIONID cookie %{JSESSIONID}C. If you are not using cookies, you can of course see the jsessionid path parameter dircetly in the logged URL. Thanks. I'll do that. (First time apache troubleshooter here). I'll get back on the results. -- Richi - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/mod_ajp-and-Load-Balancing-Issue-tp18283072p18500347.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_ajp and Load-Balancing Issue
Plana, Richard schrieb: Hi, I've two tomcat servers being proxied and load-balanced by httpd using mod_ajp (using balancer:). However, it seems when the proxy switches from one server to another, the user session gets lost. Could people recommend a way to correct my setup for doing load-balancing with this scenario? Oddly enough, it seemed to have functioned well when using mod_jk. If you don't want to replicate all sessions between your tomcat nodes, then use stickyness in the load balancer. This means, once a session has started for a user on some node, all further requests of the user belonging to this session will be routed to the same node. Give each Tomcat in server.xml a unique jvmRoute and then set the route parameter to the value of the corresponding jvmRoute of the load balancer member in your mod_proxy_balancer configuration. Look for route in the mod_proxy documentation page. Caution: the Tomcat instances need to have different jvmRoute values. The principles in mod_jk are the same. At least you need to set the jvmRoute. In mod_jk stickyness is the default, and the name of the worker I the default route. So you might have been lucky to set your jvmRoute to the worker names and that was sufficient in the mod_jk case. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: mod_ajp and Load-Balancing Issue
Hi, Thanks. Yes, if I can't share the session information between the various instances of Tomcat, I'd rather make a session sticky to a node in the load-balance pool. So I tried giving a jvmRoute to each Tomcat instance. I'm a little confused with the mod_proxy_balancer configuration, though. What I've done so far is the following: Proxy balancer://mycluster BalancerMember ajp://appserver1:8009/myapp max=10 smax=6 ttl=30 ping=120 route=jvm1 BalancerMember ajp://appserver2:8009/myapp max=10 smax=6 ttl=30 ping=120 route=jvm2 /Proxy ProxyPass /myapp balancer://mycluster/ ProxyTimeout 60 From the browser client side, I can see the .jvm[12] being appended to the JSESSIONID cookie, but it would still switch from one Tomcat to the other, albeit less frequently. What am I doing wrong? -- Richard Plana -Original Message- From: Rainer Jung [mailto:[EMAIL PROTECTED] Sent: Friday, July 04, 2008 11:13 AM To: Tomcat Users List Subject: Re: mod_ajp and Load-Balancing Issue Plana, Richard schrieb: Hi, I've two tomcat servers being proxied and load-balanced by httpd using mod_ajp (using balancer:). However, it seems when the proxy switches from one server to another, the user session gets lost. Could people recommend a way to correct my setup for doing load-balancing with this scenario? Oddly enough, it seemed to have functioned well when using mod_jk. If you don't want to replicate all sessions between your tomcat nodes, then use stickyness in the load balancer. This means, once a session has started for a user on some node, all further requests of the user belonging to this session will be routed to the same node. Give each Tomcat in server.xml a unique jvmRoute and then set the route parameter to the value of the corresponding jvmRoute of the load balancer member in your mod_proxy_balancer configuration. Look for route in the mod_proxy documentation page. Caution: the Tomcat instances need to have different jvmRoute values. The principles in mod_jk are the same. At least you need to set the jvmRoute. In mod_jk stickyness is the default, and the name of the worker I the default route. So you might have been lucky to set your jvmRoute to the worker names and that was sufficient in the mod_jk case. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_ajp and Load-Balancing Issue
Plana, Richard schrieb: Hi, Thanks. Yes, if I can't share the session information between the various instances of Tomcat, I'd rather make a session sticky to a node in the load-balance pool. So I tried giving a jvmRoute to each Tomcat instance. I'm a little confused with the mod_proxy_balancer configuration, though. What I've done so far is the following: Proxy balancer://mycluster BalancerMember ajp://appserver1:8009/myapp max=10 smax=6 ttl=30 ping=120 route=jvm1 BalancerMember ajp://appserver2:8009/myapp max=10 smax=6 ttl=30 ping=120 route=jvm2 /Proxy ProxyPass /myapp balancer://mycluster/ ProxyTimeout 60 From the browser client side, I can see the .jvm[12] being appended to the JSESSIONID cookie, but it would still switch from one Tomcat to the other, albeit less frequently. What am I doing wrong? So far, so good. You also need to tell mod_proxy_balancer, what the name of the URL parameter resp. Cookie is, which carries the routing information. I forgot about that (in mod_jk it is automatically the right for Java App Servers, mod_proxy is more flexible and you need to set it). So add stickysession=JSESSIONID|jsessionid at the end of the ProxyPass line. See also the mod_proxy docs page, look out for stcikysession. HTH Rainer Richard Plana -Original Message- From: Rainer Jung [mailto:[EMAIL PROTECTED] Sent: Friday, July 04, 2008 11:13 AM To: Tomcat Users List Subject: Re: mod_ajp and Load-Balancing Issue Plana, Richard schrieb: Hi, I've two tomcat servers being proxied and load-balanced by httpd using mod_ajp (using balancer:). However, it seems when the proxy switches from one server to another, the user session gets lost. Could people recommend a way to correct my setup for doing load-balancing with this scenario? Oddly enough, it seemed to have functioned well when using mod_jk. If you don't want to replicate all sessions between your tomcat nodes, then use stickyness in the load balancer. This means, once a session has started for a user on some node, all further requests of the user belonging to this session will be routed to the same node. Give each Tomcat in server.xml a unique jvmRoute and then set the route parameter to the value of the corresponding jvmRoute of the load balancer member in your mod_proxy_balancer configuration. Look for route in the mod_proxy documentation page. Caution: the Tomcat instances need to have different jvmRoute values. The principles in mod_jk are the same. At least you need to set the jvmRoute. In mod_jk stickyness is the default, and the name of the worker I the default route. So you might have been lucky to set your jvmRoute to the worker names and that was sufficient in the mod_jk case. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: mod_ajp and Load-Balancing Issue
-Original Message- From: Rainer Jung [mailto:[EMAIL PROTECTED] Sent: Friday, July 04, 2008 12:38 PM To: Tomcat Users List Subject: Re: mod_ajp and Load-Balancing Issue Plana, Richard schrieb: Hi, Thanks. Yes, if I can't share the session information between the various instances of Tomcat, I'd rather make a session sticky to a node in the load-balance pool. So I tried giving a jvmRoute to each Tomcat instance. I'm a little confused with the mod_proxy_balancer configuration, though. What I've done so far is the following: Proxy balancer://mycluster BalancerMember ajp://appserver1:8009/myapp max=10 smax=6 ttl=30 ping=120 route=jvm1 BalancerMember ajp://appserver2:8009/myapp max=10 smax=6 ttl=30 ping=120 route=jvm2 /Proxy ProxyPass /myapp balancer://mycluster/ ProxyTimeout 60 From the browser client side, I can see the .jvm[12] being appended to the JSESSIONID cookie, but it would still switch from one Tomcat to the other, albeit less frequently. What am I doing wrong? So far, so good. You also need to tell mod_proxy_balancer, what the name of the URL parameter resp. Cookie is, which carries the routing information. I forgot about that (in mod_jk it is automatically the right for Java App Servers, mod_proxy is more flexible and you need to set it). So add stickysession=JSESSIONID|jsessionid at the end of the ProxyPass line. See also the mod_proxy docs page, look out for stcikysession. Tried that after you mentioned it, but no, httpd still shifts my access from one node to the other (and I lose my session info). I even tried the plain example on the docs page (no jvmRoute info). Still no go. Suggestions? Ideas? -- Richi - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_ajp and Load-Balancing Issue
Plana, Richard schrieb: -Original Message- From: Rainer Jung [mailto:[EMAIL PROTECTED] Sent: Friday, July 04, 2008 12:38 PM To: Tomcat Users List Subject: Re: mod_ajp and Load-Balancing Issue Plana, Richard schrieb: Hi, Thanks. Yes, if I can't share the session information between the various instances of Tomcat, I'd rather make a session sticky to a node in the load-balance pool. So I tried giving a jvmRoute to each Tomcat instance. I'm a little confused with the mod_proxy_balancer configuration, though. What I've done so far is the following: Proxy balancer://mycluster BalancerMember ajp://appserver1:8009/myapp max=10 smax=6 ttl=30 ping=120 route=jvm1 BalancerMember ajp://appserver2:8009/myapp max=10 smax=6 ttl=30 ping=120 route=jvm2 /Proxy ProxyPass /myapp balancer://mycluster/ ProxyTimeout 60 From the browser client side, I can see the .jvm[12] being appended to the JSESSIONID cookie, but it would still switch from one Tomcat to the other, albeit less frequently. What am I doing wrong? So far, so good. You also need to tell mod_proxy_balancer, what the name of the URL parameter resp. Cookie is, which carries the routing information. I forgot about that (in mod_jk it is automatically the right for Java App Servers, mod_proxy is more flexible and you need to set it). So add stickysession=JSESSIONID|jsessionid at the end of the ProxyPass line. See also the mod_proxy docs page, look out for stcikysession. Tried that after you mentioned it, but no, httpd still shifts my access from one node to the other (and I lose my session info). I even tried the plain example on the docs page (no jvmRoute info). Still no go. Suggestions? Ideas? Is that related to the error message you posted in the parallel discussion thread, i.e. are the two things happening at the same time? If so I would say a worker went into error state because of this (and it keeps that state for some time) and all requests have been rerouted to the other worker. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: mod_ajp and Load-Balancing Issue
So far, so good. You also need to tell mod_proxy_balancer, what the name of the URL parameter resp. Cookie is, which carries the routing information. I forgot about that (in mod_jk it is automatically the right for Java App Servers, mod_proxy is more flexible and you need to set it). So add stickysession=JSESSIONID|jsessionid at the end of the ProxyPass line. See also the mod_proxy docs page, look out for stcikysession. Tried that after you mentioned it, but no, httpd still shifts my access from one node to the other (and I lose my session info). I even tried the plain example on the docs page (no jvmRoute info). Still no go. Suggestions? Ideas? Is that related to the error message you posted in the parallel discussion thread, i.e. are the two things happening at the same time? If so I would say a worker went into error state because of this (and it keeps that state for some time) and all requests have been rerouted to the other worker. Good catch, but no, it doesn't seem like the two are related. Unfortunately, I've to deal with them at the same time. The timeout that happens on the other thread usually doesn't happen for a long time (minutes). The events I'm experiencing here happen within a matter of seconds. Basically, I open a user session and go through various pages on my app. After a couple of clicks, the app just brings me back to the start page. I'm monitoring logs from both sides and I see that as soon as it comes back in error and brings me to the login page, activity switches to the other app server (the one that I wasn't using prior to the problem). Good catch, though. I've added nofailover=On to the ProxyPass line and tested it. Again, in the middle of the session, it switched to another node and I lost my session again. So not likely an error-induced failover. -- Richi - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_ajp and Load-Balancing Issue
Plana, Richard schrieb: So far, so good. You also need to tell mod_proxy_balancer, what the name of the URL parameter resp. Cookie is, which carries the routing information. I forgot about that (in mod_jk it is automatically the right for Java App Servers, mod_proxy is more flexible and you need to set it). So add stickysession=JSESSIONID|jsessionid at the end of the ProxyPass line. See also the mod_proxy docs page, look out for stcikysession. Tried that after you mentioned it, but no, httpd still shifts my access from one node to the other (and I lose my session info). I even tried the plain example on the docs page (no jvmRoute info). Still no go. Suggestions? Ideas? Is that related to the error message you posted in the parallel discussion thread, i.e. are the two things happening at the same time? If so I would say a worker went into error state because of this (and it keeps that state for some time) and all requests have been rerouted to the other worker. Good catch, but no, it doesn't seem like the two are related. Unfortunately, I've to deal with them at the same time. The timeout that happens on the other thread usually doesn't happen for a long time (minutes). The events I'm experiencing here happen within a matter of seconds. Basically, I open a user session and go through various pages on my app. After a couple of clicks, the app just brings me back to the start page. I'm monitoring logs from both sides and I see that as soon as it comes back in error and brings me to the login page, activity switches to the other app server (the one that I wasn't using prior to the problem). Just to make sure: what kind of error do you mean here? The httpd timeout and that is not the only source of lost stickyness, or some application error? If it is an application error, it would not be too strange, that the application invalidates the session. Good catch, though. I've added nofailover=On to the ProxyPass line and tested it. Again, in the middle of the session, it switched to another node and I lost my session again. So not likely an error-induced failover. If you want to debug a little more: In Tomcat you can add a %S to your log pattern, which will log the session id. In httpd you can log the Set-Cookie outgoing header %{Set-Cookie}o and the JSESSIONID cookie %{JSESSIONID}C. If you are not using cookies, you can of course see the jsessionid path parameter dircetly in the logged URL. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]