Hi Shanti, I looked and tried nginx. The imbalance is even worse. To have sticky session (for logging in and stuff), we need to use the ip_hash directive of nginx's load balancer.
"This directive causes requests to be distributed between upstreams based on the IP-address of the client. The key for the hash is the class-C network address of the client. This method guarantees that the client request will always be transferred to the same server." Since I only have 1 workload driver (hence the http requests are always from the same IP address), nginx always uses the same glassfish server. Apache mod_proxy, on the other hand, uses JSESSIONID/cookies for its sticky session. What I don't understand is even if I explicitly add calls to doLogout() in OlioDriver, there is still a big skew in the load distribution of glassfish servers. When I run it I have 1 server running with 70+% cpu utilization and 2 other servers with 40+% utilization. Simply specifying the 3 glassfish servers in OlioDriver, I get around 55-60% for each of my 3 glassfish servers. I am curious as to see whether the HttpTransport creates a new http connection or it reuses the same connection. I also read the URLConnection reuses the socket connections. "HTTPUrlConnection (not directly, but in the underlying classes) keeps a small pool of open keep-alive sockets" If HttpTransport does properly closes the connections/creates new instances, I wonder if that might explain why I'm still getting a skew when using apache mod_proxy. The thing is when I manually try accessing olio (through apache) on my browser, I do get sticky session. However, when I delete my cookies and try to access olio again, I am redirected to a different back-end server. I assume that calling HttpTransport's newInstance should clear all cookies. Thanks, Harold --- On Tue, 11/24/09, Shanti Subramanyam <[email protected]> wrote: > From: Shanti Subramanyam <[email protected]> > Subject: Re: Question about Faban/workload driver + multiple Glassfish servers > To: [email protected] > Cc: "Shanti Subramanyam" <[email protected]> > Date: Tuesday, November 24, 2009, 1:14 PM > Harold - you may also want to try > nginx. mod_proxy performance is supposed to be pretty bad. > The rails folk typically use nginx for proxying. > > Shanti > > Akara Sucharitakul wrote: > > Unfortunately not. If there is any accounting > information available through mod_proxy, this would be most > helpful. We have to determine at this level whether requests > are distributed evenly. If not, why? And can we do anything > about it, short of fixing mod_proxy. All this information is > missing. Thanks. > > > > -Akara > > > > Shanti Subramanyam wrote: > >> I assume mod_proxy will do a round-robin > scheduling of the initial request. Even if subsequent > requests in the same session are tied to the same appserver, > I would think that you should see an even distribution as > each session (on avg) should be placing the same load. > >> Let's see if Akara has better insights. > >> > >> Shanti > >> > >> Harold Lim wrote: > >>> Hi Akara, > >>> > >>> Currently, what I have is an apache front-end > using mod_proxy as a load balancer for multiple glassfish > servers. In Faban, I just enter the address of the apache > server. It works fine but I have to turn on sticky session. > Without sticky session, it's not allowing the pages to > log-in (i think because it's returning with different > glassfish server each time). > >>> > >>> The problem I have now is that when I run my > benchmark run, the load is not balanced. It is skewed > towards one of the glassfish server (i.e, 1 glassfish server > has 70+% utilization, the other 2 only has 40+%). > >>> > >>> I was looking at the UIDriver of Olio and saw > that the httpconnection instance is being reused. It only > gets "a new fresh state" in the doLogout. But, doLogout is > only being called by doLogin() and in doAddPerson(). > >>> > >>> What is the best way to address this problem? > Do I call doLogout() at the end of each benchmark operation > function (i.e, in doTagSearch(), etc)? I feel that if I do > that, it breaks the realism of the simulation/benchmark > run. > >>> > >>> In Faban, how does it determine or what > constitutes a "session". I would think a session is > something like a user going to a website, browsing a couple > of the pages within that site and then leaving. Ideally, > what we want is having something like that, with the leaving > part corresponding to calling the doLogout method. > >>> > >>> > >>> > >>> > >>> > >>> Thanks, > >>> -Harold > >>> > >>> > >>> > >>> --- On Fri, 11/6/09, Akara Sucharitakul <[email protected]> > wrote: > >>> > >>> > >>>> From: Akara Sucharitakul <[email protected]> > >>>> Subject: Re: Question about Faban/workload > driver + multiple Glassfish servers > >>>> To: "Harold Lim" <[email protected]> > >>>> Date: Friday, November 6, 2009, 4:13 PM > >>>> OK, with that version it is probably > >>>> harder to achieve what you're trying to do > (go through switches). Check out the new > >>>> benchmark class. > >>>> > >>>> -Akara > >>>> > >>>> Harold Lim wrote: > >>>> > >>>>> Hi Akara, > >>>>> > >>>>> Yes. I built my olio package from the > Java source (a > >>>>> > >>>> few months back). > >>>> > >>>>> -Harold > >>>>> --- On Fri, 11/6/09, Akara > Sucharitakul <[email protected]> > >>>>> > >>>> wrote: > >>>> > >>>>>> From: Akara Sucharitakul <[email protected]> > >>>>>> Subject: Re: Question about > Faban/workload driver > >>>>>> > > >>>> + multiple Glassfish servers > >>>> > >>>>>> To: "Harold Lim" <[email protected]> > >>>>>> Date: Friday, November 6, 2009, > 3:11 PM > >>>>>> Sorry, I meant the Olio kit. It > is > >>>>>> the Olio benchmarking process that > causes this issue. > >>>>>> > >>>>>> -Akara > >>>>>> > >>>>>> Harold Lim wrote: > >>>>>> > >>>>>>> Hi Akara, > >>>>>>> > >>>>>>> I'm using the faban binary you > gave me last > >>>>>>> > > >>>> September > >>>> > >>>>>> 19 with the load variation fix. > >>>>>> > >>>>>>> -Harold > >>>>>>> > >>>>>>> --- On Fri, 11/6/09, Akara > Sucharitakul <[email protected]> > >>>>>>> > > >>>>>> wrote: > >>>>>> > >>>>>>>> From: Akara Sucharitakul > <[email protected]> > >>>>>>>> Subject: Re: Question > about Faban/workload > >>>>>>>> > > >>>> driver > >>>> > >>>>>> + multiple Glassfish servers > >>>>>> > >>>>>>>> To: [email protected] > >>>>>>>> Date: Friday, November 6, > 2009, 2:47 PM > >>>>>>>> Hmmm, based on previous > questions I > >>>>>>>> thought you were on the > latest and pretty much in-sync with the repository. > >>>>>>>> > > >>>>>> Apparently not. > > >>>>>>>> OlioBenchmark.configure no > longer makes a > >>>>>>>> > > >>>> call to > >>>> > >>>>>> getFile. > >>>>>> > >>>>>>>> This has been eliminated > in favor of the Faban > >>>>>>>> > > >>>> services/tools > >>>> > >>>>>>>> infrastructure we had in > place lately. > >>>>>>>> > >>>>>>>> Did you build the kit > yourself? Or is this > >>>>>>>> > > >>>> based > >>>> > >>>>>> on a > >>>>>> > >>>>>>>> released kit. Thanks. > >>>>>>>> > >>>>>>>> -Akara > >>>>>>>> > >>>>>>>> Harold Lim wrote: > >>>>>>>> > > >>>>>>>>> Hi Akara, > >>>>>>>>> > >>>>>>>>> Here is the stack > trace: > >>>>>>>>> > >>>>>>>>> Logger: > >>>>>>>>> > > >>>>>> > com.sun.faban.harness.engine.GenericBenchmark > >>>>>> > >>>>>>>>> Thread > Class > > >>>>>> > > >>>>>>>> Method > >>>>>>>> > > >>>>>>>>> 11 > > >>>> > com.sun.faban.harness.engine.GenericBenchmark > >>>> > >>>>>>>> > start > >>>>>>>> > > >>>>>>>>> Exception: > >>>>>>>>> Message: > >>>>>>>>> > >>>>>>>>> > > >>>>>> > com.sun.faban.harness.FabanHostUnknownException: > >>>>>> > > >>>> Host > >>>> > >>>>>>>> xxx.xxx.xxx.xxx not > found! > >>>>>>>> > > >>>>>>>>> Stack Trace: > >>>>>>>>> Class > Method > > >>>>>> > > >>>>>>>> Line > >>>>>>>> > > >>>> com.sun.faban.harness.engine.CmdService > >>>> > >>>>>>>> > get > > >>>>>> 1276 > >>>>>> > > >>>> com.sun.faban.harness.RunContext > > >>>>>>>> getFile > 346 > >>>>>>>> > > >>>>>> > org.apache.olio.workload.harness.OlioBenchmark > >>>>>> > >>>>>>>> > configure > > >>>>>> 107 > >>>>>> > >>>>>> > > >>>> > com.sun.faban.harness.engine.InterfaceBenchmarkWrapper > >>>> > >>>>>>>> > configure > > >>>>>> 56 > >>>>>> > > >>>> > com.sun.faban.harness.engine.GenericBenchmark > >>>> > >>>>>>>> > start > > >>>>>> 264 > >>>>>> > > >>>> com.sun.faban.harness.engine.RunDaemon > >>>> > >>>>>>>> > run > > >>>>>> 338 > >>>>>> > >>>>>>>>> java.lang.Thread > > >>>> run > >>>> > >>>>>>>> > 619 > >>>>>>>> > > >>>>>>>>> Thanks! > >>>>>>>>> -Harold > >>>>>>>>> > >>>>>>>>> --- On Thu, 11/5/09, > Akara > >>>>>>>>> > > >>>> Sucharitakul <[email protected]> > >>>> > >>>>>>>> wrote: > >>>>>>>> > > >>>>>>>>>> From: Akara > Sucharitakul <[email protected]> > >>>>>>>>>> Subject: Re: > Question about > >>>>>>>>>> > > >>>> Faban/workload > >>>> > >>>>>> driver > >>>>>> > >>>>>>>> + multiple Glassfish > servers > >>>>>>>> > > >>>>>>>>>> To: [email protected] > >>>>>>>>>> Date: Thursday, > November 5, 2009, > >>>>>>>>>> > > >>>> 4:37 PM > >>>> > >>>>>>>>>> Can you get me the > stack trace? > >>>>>>>>>> Thanks. > >>>>>>>>>> > >>>>>>>>>> -Akara > >>>>>>>>>> > >>>>>>>>>> Harold Lim wrote: > >>>>>>>>>> > > >>>>>>>>>>> Hi Akara, > >>>>>>>>>>> > >>>>>>>>>>> When I set > hostConfig/enabled > >>>>>>>>>>> > > > >>>> to > >>>> > >>>>>> false, I am > >>>>>> > >>>>>>>> getting > >>>>>>>> > > >>>>>>>>>> > FabanUnknownHostException pointing > >>>>>>>>>> > > >>>> to the > >>>> > >>>>>> address > >>>>>> > >>>>>>>> of the > >>>>>>>> > > >>>>>>>>>> switch. But If I > set it to true, > >>>>>>>>>> > > >>>> it fails > >>>> > >>>>>> because > >>>>>> > >>>>>>>> it is not > >>>>>>>> > > >>>>>>>>>> able to connect to > the agent > >>>>>>>>>> > > >>>> (which I > >>>> > >>>>>> don't have) > >>>>>> > >>>>>>>> in the > >>>>>>>> > > >>>>>>>>>> machine that has > the switch. > >>>>>>>>>> > > >>>>>>>>>>> -Harold > >>>>>>>>>>> > >>>>>>>>>>> --- On Fri, > 10/30/09, Akara > >>>>>>>>>>> > > > >>>>>> Sucharitakul > >>>>>> > >>>>>>>> <[email protected]> > >>>>>>>> > > >>>>>>>>>> wrote: > >>>>>>>>>> > > >>>>>>>>>>>> From: > Akara Sucharitakul > >>>>>>>>>>>> > > > >>>> <[email protected]> > >>>> > >>>>>>>>>>>> Subject: > Re: Question > >>>>>>>>>>>> > > > >>>> about > >>>> > >>>>>> Faban/workload > >>>>>> > >>>>>>>> driver > >>>>>>>> > > >>>>>>>>>> + multiple > Glassfish servers > >>>>>>>>>> > > >>>>>>>>>>>> To: [email protected] > >>>>>>>>>>>> Date: > Friday, October 30, > >>>>>>>>>>>> > > > >>>> 2009, > >>>> > >>>>>> 7:56 PM > >>>>>> > >>>>>>>>>>>> Lets say > it has not been > >>>>>>>>>>>> > > > >>>> tested > >>>> > >>>>>> this > >>>>>> > >>>>>>>>>>>> way. It > certainly cannot > >>>>>>>>>>>> > > > >>>>>> dynamically add > >>>>>> > >>>>>>>> servers > >>>>>>>> > > >>>>>>>>>> into the > >>>>>>>>>> > > >>>>>>>>>>>> driver. > But you can use a > >>>>>>>>>>>> > > > >>>> switch > >>>> > >>>>>> that > >>>>>> > >>>>>>>> allows > >>>>>>>> > > >>>>>>>>>> dynamic > >>>>>>>>>> > > >>>>>>>>>>>> addition > of the servers. > >>>>>>>>>>>> > > > >>>> If you > >>>> > >>>>>> want > >>>>>> > >>>>>>>> stats > >>>>>>>> > > >>>>>>>>>> collection for > >>>>>>>>>> > > >>>>>>>>>>>> all the > servers behind > >>>>>>>>>>>> > > > >>>> that > >>>> > >>>>>> switch, we may > >>>>>> > >>>>>>>> need to > >>>>>>>> > > >>>>>>>>>> add an > >>>>>>>>>> > > >>>>>>>>>>>> explicit > URL field in the > >>>>>>>>>>>> > > > >>>>>> configuration > >>>>>> > >>>>>>>> (which > >>>>>>>> > > >>>>>>>>>> will default > >>>>>>>>>> > > >>>>>>>>>>>> to the web > servers if left > >>>>>>>>>>>> > > > >>>> blank. > >>>> > >>>>>> If > >>>>>> > >>>>>>>> that's not > >>>>>>>> > > >>>>>>>>>> needed, just > >>>>>>>>>> > > >>>>>>>>>>>> specify > the single server > >>>>>>>>>>>> > > > >>>>>> representing the > >>>>>> > >>>>>>>> switch > >>>>>>>> > > >>>>>>>>>> and set > >>>>>>>>>> > > >>>>>>>>>>>> > hostConfig/enabled to > >>>>>>>>>>>> > > > >>>> false to > >>>> > >>>>>> prevent > >>>>>> > >>>>>>>> Faban > >>>>>>>> > > >>>>>>>>>> starting an > >>>>>>>>>> > > >>>>>>>>>>>> agent > there. Thanks. > >>>>>>>>>>>> > >>>>>>>>>>>> -Akara > >>>>>>>>>>>> > >>>>>>>>>>>> Harold Lim > wrote: > >>>>>>>>>>>> > > > >>>>>>>>>>>>> Hi, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Does > the workload > >>>>>>>>>>>>> > > > >>>> driver > >>>> > >>>>>> supports > >>>>>> > >>>>>>>> dynamically > >>>>>>>> > > >>>>>>>>>> adding a > >>>>>>>>>> > > >>>>>>>>>>>> glassfish > server, while a > >>>>>>>>>>>> > > > >>>>>> benchmark test > >>>>>> > >>>>>>>> is > >>>>>>>> > > >>>>>>>>>> running? > >>>>>>>>>> > > >>>>>>>>>>>>> > Currently, I specify > >>>>>>>>>>>>> > > > >>>> all the > >>>> > >>>>>> glassfish > >>>>>> > >>>>>>>> servers > >>>>>>>> > > >>>>>>>>>> in the > >>>>>>>>>> > > >>>>>>>>>>>> host:pair > field of the web > >>>>>>>>>>>> > > > >>>> server > >>>> > >>>>>> tab. > >>>>>> > >>>>>>>>>>>>> If > not, is there any > >>>>>>>>>>>>> > > > >>>> way > >>>> > >>>>>> around it? Do > >>>>>> > >>>>>>>> you > >>>>>>>> > > >>>>>>>>>> have any > >>>>>>>>>> > > >>>>>>>>>>>> ideas of > how to deal with > >>>>>>>>>>>> > > > >>>> adding > >>>> > >>>>>> or > >>>>>> > >>>>>>>> removing > >>>>>>>> > > >>>>>>>>>> glassfish > >>>>>>>>>> > > >>>>>>>>>>>> servers > dynamically? > >>>>>>>>>>>> > > > >>>>>>>>>>>>> > -Harold > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > > > >>>> > >>>>>>>> > > >>>>>>> > >>>>>>> > > >>>>> > >>>>> > >>>> > >>> > >>> > >>> > >>> > > > >
