Ravindra, Thx for idea, I will read about it, but at first glance it looks like with 5000 pending servlet requests I will have 5000 threads awaiting response from cxf client, with async servlet and async cxf webservice server, connected with async cxf client, all 5000 pending threads can be served by one thread,
I have to deal not only with faults, but also with long running webservice client requests - up to two client requests, 20 second each regards Jakub ps possibly apache camel can do with it, I have to check in spare time. On Mon, Jun 17, 2013 at 5:51 AM, Vanga Palli, Ravindra Kumar < ravindra.vangapa...@hp.com> wrote: > Ja kub, > > Looks like you are re-inventing wheel here. All you are looking for is a > fault tolerance system, you should consider exploring hystrix - latency > and fault tolerance for distributed systems library. > > https://github.com/Netflix/Hystrix > > -Ravi > > ________________________________________ > From: Ja kub [jjaku...@gmail.com] > Sent: Thursday, June 13, 2013 1:11 AM > To: Tomcat Users List > Subject: Re: http request (no only session) replication in cluster > > Christopher > Thx for response, I will inform guys from business about what You have > written, and let them consider it > > Regards > Jakub > > > On Wed, Jun 12, 2013 at 4:10 PM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA256 > > > > Jakob, > > > > On 6/11/13 5:04 PM, Ja kub wrote: > > > requirement is system should be possible to process 160 req/sec > > > (200 is better to multiply) and system is kind of failover proxy > > > itself > > > > > > there are 2 backing webservices, each can answer max 20s, it there > > > is timeout on first, I must call the second, if there is timeout on > > > second I send soap fault to client, so usually it shouldn't be more > > > than 20s per req, guys say that normally it is 7-10 > > > seconds/request, but in worst case it is 2*20s*160 requests/s ~= > > > 6400 pending requests (and according to deal we must fulfill worst > > > case) > > > > If you have 2 member nodes and one of them starts to slow down, then > > you'll see pretty much all requests re-tried on the second node, which > > will slow down that one. I think you'll end up seeing a storm of > > requests bouncing back and forth. > > > > Worse, the initial request will continue processing on the 1st node, > > ignorant of the fact that the lb has given up and tried the other > > node. It's just going to fall apart from there. > > > > Honestly, this should be able to be handled at your lb -- can't you > > set a time-out there? > > > > > even if there are so many requests they are pending on sockets, I > > > try to do it with NIO, asynchronous servlets and async cxf - both > > > async cxf webservice is exposed by me, and I also call backing ws > > > with async cxf I think even one tomcat on one server should be able > > > to serve such 6400 pending requests with 160req/s, apart from proxy > > > there are also 4-6 inserts into database (cli req, resp; 1st ws > > > call, resp; 2nd ws call, resp > > > > > > how do You assess such architecture/attitude ? do You expect > > > problems with async exposed webservice based on async servlet and > > > NIO, and async cxf ws client ? afaik cxf use thread locals, are > > > they all right with tomcat async servlets ? (I don't define > > > threadlocals by myself, only cxf possibly does) > > > > It's not a socket-resource issue, it's a raw work-load issue: you have > > a large amount of load and it looks like you can't handle it very > > well. I would recommend more nodes, first, and then seriously consider > > whether re-trying on a second node is appropriate if the first node > > takes too long. > > > > What you should probably do is actually profile your code to find out > > what is taking so long. Using tricks like ThreadLocals can shed > > microseconds off of a request, not whole seconds. > > > > You might want to consider whether you can do less work during a > > request -- perhaps split a single transaction into more than one. Or, > > just acknowledge that sometimes a transaction can take 10-20 seconds > > (or 50?) and manage the clients' expectations. > > > > You also need to find out where your bottleneck is: RDBMSs, slow > > disks, slow network links, etc. can all be much more significant than > > things like software stack and exact implementation of your code. If > > you are missing an index on a relational table, transactions that > > should take a second or two can take tens of seconds. > > > > Start there: profile your application, find out what is slow, and fix > > that. Don't try to work-around the problem with surprising > > transactional re-tries, because they likely won't work the way you > > hoped. Hey, once you fix your performance problem, perhaps you won't > > need additional hardware. Also, your users will be very happy to see a > > speed improvement. > > > > - -chris > > -----BEGIN PGP SIGNATURE----- > > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > > Comment: GPGTools - http://gpgtools.org > > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > > > iQIcBAEBCAAGBQJRuIFyAAoJEBzwKT+lPKRYr+4QAL7UCQvZ/CBIueIqhFDLkZ57 > > v0uWcuukEtT8/8dNUBW2SGSE4OwDyH41Nsx7ZVo4W+lTnzduVbjXSvU4lXNDiY19 > > 9MgpwuxZWlUxHAgOQ5NODIFwrHQK2GYMe8+Qo8OBVf6lOhVdB4PS/7XM7lrsnWBn > > rpojl4rPm7+esciZPB1q15+PxCbgh4uGsI4KCXyZiW/gz/dLC2v8u6QiYfqoXgov > > iutYtII+7f1E6I+Ag3LjmQwrzY7pRpHrotcJ4aCpyOs9EHTavKf3mapwY2HiOP+t > > G9qwGuq5tUJhkBzF5Vdvqf+lCbdJHkQtLW3Z4vL4/XTK7SVSvjipFhsttZF4TII6 > > 6QVQmjCJZRYdPDegzB+NVaCxPkdZLLdwHNFFfsZGabdTQDkAKOEXQiYjBqJ9n5nX > > WRHvYLQtyGEj1e+0zqwCihRHie2TbfwdggtCoVaOF+8Zpguv3K9VRHwvFA/miA1i > > JkYCfxKjyF/RoCyB4wZqCi5VsJjztQpq6uDQiUG0CACY1491sB35M+Vkqm3jqRbh > > 0HXs1ckqZsw+2Y013kpCVs0eipOst5GD6XqXr6LTT/fQwEYWa3uVTk3/h2xDd9BT > > DlTZrs1CNhqMBjNqUDUFkiiempf9kFkQhrao50CAilix95/VhdWkDjFcFSKKQ0/J > > EkcONNIioMTN7cWzKNHf > > =miI6 > > -----END PGP SIGNATURE----- > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
