RE: [squid-users] squid crashes by itself and reboots automatically
From: To: gkin...@gmail.com CC: squid-users@squid-cache.org Date: Mon, 30 Nov 2009 07:15:27 +0200 Subject: RE: [squid-users] squid crashes by itself and reboots automatically The only entry that I have in the cache.log are these lines before that I have all storelocatevary warnings but nothing out of the ordinary. FATAL: Received Segment Violation...dying. 2009/11/29 22:45:48| storeDirWriteCleanLogs: Starting... fatal seg fault is your strcmp throw? I guess null pointer but could just be passing code or non-zero junk. 2009/11/29 22:45:48| WARNING: Closing open FD 24 2009/11/29 22:45:48| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed on fd=24: (1) Operation not permitted Lines that are before this error are all similar to these This sounds like it has an invalid object, which would suggest a reason for passing a silly param to strcmp? storeLocateVary: Not our vary marker object, 9884D2F9295EC65D66A36B329A3C7BBD = 'http://b.static.ak.fbcdn.net/rsrc.php/zE010/hash/2kkc6o 96.css', 'accept-encoding=gzip,%20deflate'/'gzip, deflate' 2009/11/29 22:45:47| storeLocateVary: Not our vary marker object, DCBAB677A2364853342DA79508301881 = 'http://static.ak.fbcdn.net/rsrc.php/z5WTN/hash/arvryq2p .css', 'accept-encoding=gzip,%20deflate'/'gzip, deflate' 2009/11/29 22:45:47| storeLocateVary: Not our vary marker object, FBD3324530E5C7D93199529A191361C0 = 'http://static.ak.fbcdn.net/rsrc.php/zBICQ/hash/ryfrvghz .css', 'accept-encoding=gzip,deflate,sdch'/'gzip,deflate,sdch' 2009/11/29 22:45:47| storeLocateVary: Not our vary marker object, 498DE9ABC02057F4C59D6A60B2E515B1 = 'http://static.ak.fbcdn.net/rsrc.php/z37DG/hash/e5d2uja2 .css', 'accept-encoding=gzip,deflate,sdch'/'gzip,deflate,sdch' 2009/11/29 22:45:47| storeLocateVary: Not our vary marker object, 956DBDC557535910ECC2BDE74EEC30D4 = 'http://b.static.ak.facebook.com/common/redirectiframe.h tml', 'accept-encoding=gzip,%20deflate'/'gzip, deflate' Sincerely, Ragheb Rustom -Original Message- From: Kinkie [mailto:gkin...@gmail.com] Sent: Monday, November 30, 2009 1:25 AM To: Ragheb Rustom Cc: squid-users@squid-cache.org Subject: Re: [squid-users] squid crashes by itself and reboots automatically On Sun, Nov 29, 2009 at 8:25 PM, Ragheb Rustom wrote: Dear all, I have multiple servers running squid 2.7-stable7. Lately I have noticed that most of these servers are crashing sometimes everyday creating a core dump file and then after creating the file restart by itself. After reading the core file with gdb and doing a backtrace I got the following info. Can anyone please help me identify what is going on with these squid servers. You help is very valuable. Is there anything in cache.log? Possibly just the few lines before the restart. -- /kinkie _ Hotmail: Trusted email with Microsoft's powerful SPAM protection. http://clk.atdmt.com/GBL/go/177141664/direct/01/ http://clk.atdmt.com/GBL/go/177141664/direct/01/
Re: [squid-users] squid crashes by itself and reboots automatically
On Sun, Nov 29, 2009 at 8:25 PM, Ragheb Rustom rag...@smartelecom.org wrote: Dear all, I have multiple servers running squid 2.7-stable7. Lately I have noticed that most of these servers are crashing sometimes everyday creating a core dump file and then after creating the file restart by itself. After reading the core file with gdb and doing a backtrace I got the following info. Can anyone please help me identify what is going on with these squid servers. You help is very valuable. Is there anything in cache.log? Possibly just the few lines before the restart. -- /kinkie
RE: [squid-users] squid crashes by itself and reboots automatically
Date: Mon, 30 Nov 2009 00:24:38 +0100 From: gkin...@gmail.com To: rag...@smartelecom.org CC: squid-users@squid-cache.org Subject: Re: [squid-users] squid crashes by itself and reboots automatically On Sun, Nov 29, 2009 at 8:25 PM, Ragheb Rustom wrote: Dear all, I have multiple servers running squid 2.7-stable7. Lately I have noticed that most of these servers are crashing sometimes everyday creating a core dump file and then after creating the file restart by itself. After reading the core file with gdb and doing a backtrace I got the following info. Can anyone please help me identify what is going on with these squid servers. You help is very valuable. Is there anything in cache.log? Possibly just the few lines before the restart. the default config file seems good about logging stuff. Was that sparse stack trace typical or just the first one you had? The only sig that was obvious was strcmp- if it was blowing up there you either had pathological strings or memory corruption or timing issue etc. Not too informative until you at least know if it is a consistent failure point. -- /kinkie _ Windows 7: I wanted simpler, now it's simpler. I'm a rock star. http://www.microsoft.com/Windows/windows-7/default.aspx?h=myidea?ocid=PID24727::T:WLMTAGL:ON:WL:en-US:WWL_WIN_myidea:112009
RE: [squid-users] squid crashes by itself and reboots automatically
The only entry that I have in the cache.log are these lines before that I have all storelocatevary warnings but nothing out of the ordinary. FATAL: Received Segment Violation...dying. 2009/11/29 22:45:48| storeDirWriteCleanLogs: Starting... 2009/11/29 22:45:48| WARNING: Closing open FD 24 2009/11/29 22:45:48| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed on fd=24: (1) Operation not permitted Lines that are before this error are all similar to these storeLocateVary: Not our vary marker object, 9884D2F9295EC65D66A36B329A3C7BBD = 'http://b.static.ak.fbcdn.net/rsrc.php/zE010/hash/2kkc6o 96.css', 'accept-encoding=gzip,%20deflate'/'gzip, deflate' 2009/11/29 22:45:47| storeLocateVary: Not our vary marker object, DCBAB677A2364853342DA79508301881 = 'http://static.ak.fbcdn.net/rsrc.php/z5WTN/hash/arvryq2p .css', 'accept-encoding=gzip,%20deflate'/'gzip, deflate' 2009/11/29 22:45:47| storeLocateVary: Not our vary marker object, FBD3324530E5C7D93199529A191361C0 = 'http://static.ak.fbcdn.net/rsrc.php/zBICQ/hash/ryfrvghz .css', 'accept-encoding=gzip,deflate,sdch'/'gzip,deflate,sdch' 2009/11/29 22:45:47| storeLocateVary: Not our vary marker object, 498DE9ABC02057F4C59D6A60B2E515B1 = 'http://static.ak.fbcdn.net/rsrc.php/z37DG/hash/e5d2uja2 .css', 'accept-encoding=gzip,deflate,sdch'/'gzip,deflate,sdch' 2009/11/29 22:45:47| storeLocateVary: Not our vary marker object, 956DBDC557535910ECC2BDE74EEC30D4 = 'http://b.static.ak.facebook.com/common/redirectiframe.h tml', 'accept-encoding=gzip,%20deflate'/'gzip, deflate' Sincerely, Ragheb Rustom -Original Message- From: Kinkie [mailto:gkin...@gmail.com] Sent: Monday, November 30, 2009 1:25 AM To: Ragheb Rustom Cc: squid-users@squid-cache.org Subject: Re: [squid-users] squid crashes by itself and reboots automatically On Sun, Nov 29, 2009 at 8:25 PM, Ragheb Rustom rag...@smartelecom.org wrote: Dear all, I have multiple servers running squid 2.7-stable7. Lately I have noticed that most of these servers are crashing sometimes everyday creating a core dump file and then after creating the file restart by itself. After reading the core file with gdb and doing a backtrace I got the following info. Can anyone please help me identify what is going on with these squid servers. You help is very valuable. Is there anything in cache.log? Possibly just the few lines before the restart. -- /kinkie
RE: [squid-users] squid crashes by itself and reboots automatically
in ?? () #9 0x004a8fca in aioCheckCallbacks () #10 0x00493137 in ?? () #11 0x00438b0e in ?? () #12 0x00469d6d in ?? () #13 0x00393181e32a in __libc_start_main () from /lib64/libc.so.6 #14 0x00409cb9 in drand48 () #15 0x7fffdec70c78 in ?? () #16 0x001c in ?? () #17 0x0002 in ?? () #18 0x7fffdec70e92 in ?? () #19 0x7fffdec70e9a in ?? () #20 0x in ?? () (gdb) Ragheb Rustom Smartelecom S.A.R.L Sin el fil - Mar Elias Street Absi Center - Ground Floor Beirut, Lebanon Telefax: +961-1-487275 Mobile: +961-3-286282 Email: rag...@smartelecom.org -Original Message- From: Mike Marchywka [mailto:marchy...@hotmail.com] Sent: Monday, November 30, 2009 1:50 AM To: gkin...@gmail.com; rag...@smartelecom.org Cc: squid-users@squid-cache.org Subject: RE: [squid-users] squid crashes by itself and reboots automatically Date: Mon, 30 Nov 2009 00:24:38 +0100 From: gkin...@gmail.com To: rag...@smartelecom.org CC: squid-users@squid-cache.org Subject: Re: [squid-users] squid crashes by itself and reboots automatically On Sun, Nov 29, 2009 at 8:25 PM, Ragheb Rustom wrote: Dear all, I have multiple servers running squid 2.7-stable7. Lately I have noticed that most of these servers are crashing sometimes everyday creating a core dump file and then after creating the file restart by itself. After reading the core file with gdb and doing a backtrace I got the following info. Can anyone please help me identify what is going on with these squid servers. You help is very valuable. Is there anything in cache.log? Possibly just the few lines before the restart. the default config file seems good about logging stuff. Was that sparse stack trace typical or just the first one you had? The only sig that was obvious was strcmp- if it was blowing up there you either had pathological strings or memory corruption or timing issue etc. Not too informative until you at least know if it is a consistent failure point. -- /kinkie _ Windows 7: I wanted simpler, now it's simpler. I'm a rock star. http://www.microsoft.com/Windows/windows-7/default.aspx?h=myidea?ocid=PID247 27::T:WLMTAGL:ON:WL:en-US:WWL_WIN_myidea:112009
Re: [squid-users] squid crashes by itself and reboots automatically
Ragheb Rustom wrote: Hi Mike, These are the gdb bt of the core dumps that I got on other servers showing the same behavior. It seems that are all failing at the strcmp. Any ideas what could be causing this? Not without debug symbols to identify the area of code calling strcmp(). There are just too many places its used for guesses to be possible. The cache.log is usually helpful, but the messages you posted earlier appear to be nowhere near any strcmp(). You could try the daily snapshot with debug symbols left in and see what appears. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 Current Beta Squid 3.1.0.15
