Shanti, After going over the commons-httpclient code I believe that the client is behaving as expected and the server for some reason stops sending data, while the clients expects them. I am planning to upgrade my apache version to 2.2.15, so that I get all the latest updates and bug fixes. (and so that I can go over the related code as well)
So again sanity check, since I got no reply on this one... I am currently using: *php* v.5.2.4 (without Suhosin patch, I removed it) *Server* version: Apache/2.2.11 (Ubuntu) *memcached* 1.2.2* (*from phpinfo() I read *memcache *2.2.5) *mysql* Ver 14.12 Distrib 5.0.75, for debian-linux-gnu (x86_64) *apc* 3.0.19 Shanti/Akara, can you tell me the versions you are using for testing? Just to be sure I am on the same page. -Thanks ------------------------------------------------------------------- Kontorinis Vasileios Phd student, University of California San Diego http://cseweb.ucsd.edu/~vkontori/ [email protected] ------------------------------------------------------------------- 2010/5/24 Vasileios Kontorinis <[email protected]> > Well, > It feels like there are more than one things wrong in my configuration. > I stopped using the thumper and went back to using the local fs of the > web-server. > > Some of my issues went away but not all of them. > Now it fails less often, every 5-10 runs. > > The error message in the faban logs: > > *UIDriverAgent[0].752.do <op> : chunked stream ended unexpectedly > Note: Error not counted in result. > Either transaction start or end time is not within steady state. > * > (if the error happens in the steady state it is counted...) > > I looked a little bit about it online, it seems like the server is getting > a response that is not expecting. I have no clue how to debug that. > Any ideas? > > Just a sanity check. I am using: > *php* v.5.2.4 (without Suhosin patch, I removed it) > *Server* version: Apache/2.2.11 (Ubuntu) > *memcached* 1.2.2* (*from phpinfo() I read *memcache *2.2.5) > *mysql* Ver 14.12 Distrib 5.0.75, for debian-linux-gnu (x86_64) > *apc* 3.0.19 > > Is this old?? I am considering upgrading my Ubuntu version from Hardy to > Lucid with all the related package updates. > As far php is concerned I know the current version is 5.2.12 so mine is > pretty old. > > Let me know > Thanks again. > > ------------------------------------------------------------------- > Kontorinis Vasileios > Phd student, University of California San Diego > http://cseweb.ucsd.edu/~vkontori/ <http://cseweb.ucsd.edu/%7Evkontori/> > [email protected] > ------------------------------------------------------------------- > > > 2010/5/24 Vasileios Kontorinis <[email protected]> > > I am using v3. >> >> What I observed and made me suspicious of nfs is in syslog a bunch of >> messages: >> May 22 00:22:37 olio-web -- MARK -- >> May 22 00:30:29 olio-web kernel: [2167206.713881] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:03 olio-web kernel: [2167240.993893] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:06 olio-web kernel: [2167243.193896] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:07 olio-web kernel: [2167244.349889] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:07 olio-web kernel: [2167244.357893] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:07 olio-web kernel: [2167244.669885] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:10 olio-web kernel: [2167247.781891] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:10 olio-web kernel: [2167247.785889] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:11 olio-web kernel: [2167248.725885] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:13 olio-web kernel: [2167250.153892] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:14 olio-web kernel: [2167251.173886] nfs: server 67.58.51.149 >> not responding, still trying >> May 22 00:31:14 olio-web kernel: [2167251.410700] nfs: server 67.58.51.149 >> OK >> May 22 00:31:14 olio-web kernel: [2167251.411158] nfs: server 67.58.51.149 >> OK >> May 22 00:31:14 olio-web kernel: [2167251.411236] nfs: server 67.58.51.149 >> OK >> May 22 00:31:14 olio-web kernel: [2167251.411249] nfs: server 67.58.51.149 >> OK >> May 22 00:31:14 olio-web kernel: [2167251.411354] nfs: server 67.58.51.149 >> OK >> May 22 00:31:14 olio-web kernel: [2167251.411678] nfs: server 67.58.51.149 >> OK >> May 22 00:31:14 olio-web kernel: [2167251.411691] nfs: server 67.58.51.149 >> OK >> May 22 00:31:14 olio-web kernel: [2167251.411712] nfs: server 67.58.51.149 >> OK >> May 22 00:31:14 olio-web kernel: [2167251.411723] nfs: server 67.58.51.149 >> OK >> May 22 00:31:14 olio-web kernel: [2167251.412047] nfs: server 67.58.51.149 >> OK >> May 22 00:31:14 olio-web kernel: [2167251.414051] nfs: server 67.58.51.149 >> OK >> May 22 00:42:37 olio-web -- MARK -- >> May 22 01:02:37 olio-web -- MARK -- >> >> Regarding the olio run log it's weird. >> Usually says nothing. Sometimes it gives a bunch of >> >> UIDriverAgent[0].752.do <op> : chunked stream ended unexpectedly >> Note: Error not counted in result. >> Either transaction start or end time is not within steady state. >> >> and also >> >> UIDriverAgent[0].635.do <op> : connect timed out >> Note: Error not counted in result. >> Either transaction start or end time is not within steady state. >> >> where <op> different operations.... >> >> But it does not log this every time... >> >> Is there a way to monitor nfs internals (buffering, throughput etc) on the >> fly? >> I am using sar but it does not give much helpful info? >> >> >> ------------------------------------------------------------------- >> Kontorinis Vasileios >> Phd student, University of California San Diego >> http://cseweb.ucsd.edu/~vkontori/ <http://cseweb.ucsd.edu/%7Evkontori/> >> [email protected] >> ------------------------------------------------------------------- >> >> >> 2010/5/24 Shanti Subramanyam <[email protected]> >> >> Which version of NFS are you using ? I suggest you try v3 - it is more >>> efficient. We've run into issues with v4 causing unacceptable response >>> times. >>> For the kind of drop you are seeing, I am surprised that you don't find >>> any errors anywhere. In the first 400 user run for example, it looks like >>> all the processes either exited or are stalled. In either case, I would >>> expect to see errors in the faban run log (either that the driver got an >>> error or that it timed out). Are you sure you checked the faban log ? >>> >>> Have you tried running nfsstat to see if you can spot anything ? >>> >>> Shanti >>> >>> >>> On Sun, May 23, 2010 at 7:07 PM, Vasileios Kontorinis < >>> [email protected]> wrote: >>> >>>> Shanti hi again, >>>> I sort of managed to fix that. I tried upgrading my php version to >>>> 5.2.6 and the alert went away. My problems though are not fixed. >>>> I even tried completely removing suhosin patch (it was a huge pain in >>>> ubuntu, since you need to recompile the php module by yourself) >>>> Still though my proms are there. >>>> >>>> Now, I get no warning the logs are clean but I get weird behavior. I >>>> needed to send you guys some pics so I created a related page at: >>>> http://cseweb.ucsd.edu/~vkontori/olio/olio.html<http://cseweb.ucsd.edu/%7Evkontori/olio/olio.html> >>>> I have comments describing the prom at the end. >>>> >>>> <http://cseweb.ucsd.edu/%7Evkontori/olio/olio.html>Any help would be >>>> most appreciated. I ve spent so much time on it without figuring it out. >>>> My configuration is 1 web server on a vm with 6GB of mem. 4 cpus >>>> 1 db server on a vm with 5GB of mem. 4 >>>> cpus >>>> 1 fs server on a vm with 4GB of mem. 4 >>>> cpus. (this one just exposes over NFS the filestore) >>>> All on the same physical machine a nehalem based server, siting on a >>>> Sun's Black box. >>>> I got similar behavior when I exposed the filestore on the Sun's >>>> thumper. >>>> >>>> Any help would be most appreciated. >>>> >>>> Thanks >>>> >>>> ------------------------------------------------------------------- >>>> Kontorinis Vasileios >>>> Phd student, University of California San Diego >>>> http://cseweb.ucsd.edu/~vkontori/ <http://cseweb.ucsd.edu/%7Evkontori/> >>>> [email protected] >>>> ------------------------------------------------------------------- >>>> >>>> >>>> 2010/5/19 Shanti Subramanyam <[email protected]> >>>> >>>> It's strange that multiple files seem to be complaining about it. Did >>>>> you try disabling Suhosin ? Are you seeing a perceptible drop in memory >>>>> after reaching steady state ? >>>>> >>>>> shanti >>>>> >>>>> >>>>> On Wed, May 19, 2010 at 4:28 PM, Vasileios Kontorinis < >>>>> [email protected]> wrote: >>>>> >>>>>> Lately I get a bunch of these errors in my logs: >>>>>> >>>>>> [Wed May 19 22:26:37 2010] [error] [client 10.17.255.250] ALERT - >>>>>> canary mismatch on efree() - heap overflow detected (attacker >>>>>> '10.17.255.250', file '/var/www/oliophp/public_html/taggedEvents.php') >>>>>> [Wed May 19 22:26:37 2010] [error] [client 10.17.255.250] ALERT - >>>>>> canary mismatch on efree() - heap overflow detected (attacker >>>>>> '10.17.255.250', file '/var/www/oliophp/public_html/taggedEvents.php') >>>>>> [Wed May 19 22:26:37 2010] [error] [client 10.17.255.250] ALERT - >>>>>> canary mismatch on efree() - heap overflow detected (attacker >>>>>> '10.17.255.250', file '/var/www/oliophp/public_html/users.php') >>>>>> [Wed May 19 22:26:37 2010] [error] [client 10.17.255.250] ALERT - >>>>>> canary mismatch on efree() - heap overflow detected (attacker >>>>>> '10.17.255.250', file '/var/www/oliophp/public_html/events.php') >>>>>> [Wed May 19 22:26:37 2010] [error] [client 10.17.255.250] ALERT - >>>>>> canary mismatch on efree() - heap overflow detected (attacker >>>>>> '10.17.255.250', file '/var/www/oliophp/public_html/taggedEvents.php') >>>>>> >>>>>> According to blogs it is a php related issue. Suhosin patch detects a >>>>>> memory overflow and complains. >>>>>> I was just wondering if the Olio php code is having any known mem. >>>>>> leaks. >>>>>> >>>>>> My php version on ubuntu: >>>>>> PHP 5.2.4-2ubuntu5 with Suhosin-Patch 0.9.6.2 (cli) (built: Feb 27 >>>>>> 2008 20:46:51) >>>>>> Copyright (c) 1997-2007 The PHP Group >>>>>> Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies >>>>>> >>>>>> It's too bad that I do not get a line on the php files that cause >>>>>> this. >>>>>> >>>>>> >>>>>> Has anyone come across this one before? >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------- >>>>>> Kontorinis Vasileios >>>>>> Phd student, University of California San Diego >>>>>> San Diego, CA 92122 >>>>>> Cell. phone: (858) 717 6899 >>>>>> [email protected], [email protected] >>>>>> ------------------------------------------------------------------- >>>>>> >>>>> >>>>> >>>> >>> >> >
