Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

Okie dokie! boxes are crashing all over the place today so I finally have
some back traces without stuff optimised out.

Here are the details from two of these crashes which occurred on two
separate deployments—please let me know if they actually contain actionable
information now and I will upload them to the bug.

Thanks folks.


On 25 March 2015 at 09:28, Dan Charlesworth d...@getbusi.com wrote:

 Resending this after the last attempt went into the mail server black hole:

 Hey Amos

 I decided I’m not confident enough in 3.5.HEAD, after last time, to go
 back into production with it. Going to to do some more local testing first.

 That being said, I now have 3.4.12 in production with optimisations
 disabled and it seems to be doing fine performance and stability-wise. I
 only managed to capture one crash with optimisations disabled, so far, but
 it seemed to have some memory-related corruption, unfortunately.

 Updates to come over the next few days.


 On 23 March 2015 at 16:59, Dan Charlesworth d...@getbusi.com wrote:

 Hey Amos

 I decided I’m not confident enough in 3.5.HEAD, after last time, to go
 back into production with it. Going to to do some more local testing first.

 That being said, I now have 3.4.12 in production with optimisations
 disabled and it seems to be doing fine performance and stability-wise. I
 only managed to capture one crash with optimisations disabled, so far, but
 it seemed to have some memory-related corruption, unfortunately.

 More to come tomorrow :-)

  On 20 Mar 2015, at 6:37 pm, Amos Jeffries squ...@treenet.co.nz wrote:
 
  On 20/03/2015 8:34 p.m., Dan Charlesworth wrote:
  Thanks Amos.
 
 
  I'll put together a build with the upcoming snapshot on Monday, might
 even try disabling optimization for it too.
 
  Please do. If you're only getting 40 RPS out of the proxy during the
  test its hard to see how not optimizing the code could be any worse, and
  it will help identifiying some traffic details.
 
  Amos
 



#0  0x003edf832625 in raise (sig=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x003edf833e05 in abort () at abort.c:92
#2  0x0062105f in xassert (msg=0x91a311 
connIsUsable(http-getConn()), file=0x9198b0 client_side.cc, line=1515) at 
debug.cc:566
#3  0x005db79f in clientSocketRecipient (node=0x50f0d008, 
http=0x51075ab8, rep=0x12f95940, receivedData=...) at client_side.cc:1515
#4  0x0061b574 in clientStreamCallback (thisObject=0x451353c8, 
http=0x51075ab8, rep=0x12f95940, replyBuffer=...) at clientStream.cc:186
#5  0x006061eb in clientReplyContext::processReplyAccessResult 
(this=0x5118d098, accessAllowed=...) at client_side_reply.cc:2058
#6  0x006057fd in clientReplyContext::ProcessReplyAccessResult (rv=..., 
voidMe=0x5118d098) at client_side_reply.cc:1961
#7  0x007cd727 in ACLChecklist::checkCallback (this=0x573cd338, 
answer=...) at Checklist.cc:161
#8  0x007ccddb in ACLChecklist::completeNonBlocking (this=0x573cd338) 
at Checklist.cc:46
#9  0x007cddd6 in ACLChecklist::resumeNonBlockingCheck 
(this=0x573cd338, state=0xc6ca20) at Checklist.cc:279
#10 0x0064cc01 in ExternalACLLookup::LookupDone (data=0x573cd338, 
result=...) at external_acl.cc:1623
#11 0x0064bd37 in externalAclHandleReply (data=0x573a3918, reply=...) 
at external_acl.cc:1427
#12 0x0067e01f in helperReturnBuffer (request_number=0, srv=0x17021e8, 
hlp=0x1701fe8, 
msg=0x17023a0 ERR 
log=%7B%22policy_group_id%22%3A%226%22%2C%22categories%22%3A%22%5B28%5D%22%2C%22user%22%3A%2215ifrain%22%2C%22set_id%22%3A%222%22%2C%22user_group%22%3A%22stu2015%22%7D,
 msg_end=0x170244b ) at helper.cc:858
#13 0x0067e9f3 in helperHandleRead (conn=..., 
buf=0x17023a0 ERR 
log=%7B%22policy_group_id%22%3A%226%22%2C%22categories%22%3A%22%5B28%5D%22%2C%22user%22%3A%2215ifrain%22%2C%22set_id%22%3A%222%22%2C%22user_group%22%3A%22stu2015%22%7D,
 len=172, flag=COMM_OK, xerrno=0, data=0x17021e8) at helper.cc:951
#14 0x007e6f2a in CommIoCbPtrFun::dial (this=0x4d838d80) at 
CommCalls.cc:188
#15 0x005fb498 in CommCbFunPtrCallTCommIoCbPtrFun::fire 
(this=0x4d838d50) at CommCalls.h:376
#16 0x007d2b40 in AsyncCall::make (this=0x4d838d50) at AsyncCall.cc:32
#17 0x007d61ff in AsyncCallQueue::fireNext (this=0x15e4c60) at 
AsyncCallQueue.cc:52
#18 0x007d5f5f in AsyncCallQueue::fire (this=0x15e4c60) at 
AsyncCallQueue.cc:38
#19 0x00644bef in EventLoop::dispatchCalls (this=0x7fffad41f1c0) at 
EventLoop.cc:158
#20 0x00644a7f in EventLoop::runOnce (this=0x7fffad41f1c0) at 
EventLoop.cc:135
#21 0x006448b8 in EventLoop::run (this=0x7fffad41f1c0) at 
EventLoop.cc:99
#22 0x006cddbe in SquidMain (argc=3, argv=0x7fffad41f3f8) at 
main.cc:1528
#23 0x006cd32d in SquidMainSafe (argc=3, argv=0x7fffad41f3f8) at 
main.cc:1260
#24 0x006cd308 in main (argc=3, argv=0x7fffad41f3f8) at main.cc:1252
#0  0x003edf832625 in raise (sig=6) at 

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

Resending this after the last attempt went into the mail server black hole:

Hey Amos

I decided I’m not confident enough in 3.5.HEAD, after last time, to go back
into production with it. Going to to do some more local testing first.

That being said, I now have 3.4.12 in production with optimisations
disabled and it seems to be doing fine performance and stability-wise. I
only managed to capture one crash with optimisations disabled, so far, but
it seemed to have some memory-related corruption, unfortunately.

Updates to come over the next few days.


On 23 March 2015 at 16:59, Dan Charlesworth d...@getbusi.com wrote:

 Hey Amos

 I decided I’m not confident enough in 3.5.HEAD, after last time, to go
 back into production with it. Going to to do some more local testing first.

 That being said, I now have 3.4.12 in production with optimisations
 disabled and it seems to be doing fine performance and stability-wise. I
 only managed to capture one crash with optimisations disabled, so far, but
 it seemed to have some memory-related corruption, unfortunately.

 More to come tomorrow :-)

  On 20 Mar 2015, at 6:37 pm, Amos Jeffries squ...@treenet.co.nz wrote:
 
  On 20/03/2015 8:34 p.m., Dan Charlesworth wrote:
  Thanks Amos.
 
 
  I'll put together a build with the upcoming snapshot on Monday, might
 even try disabling optimization for it too.
 
  Please do. If you're only getting 40 RPS out of the proxy during the
  test its hard to see how not optimizing the code could be any worse, and
  it will help identifiying some traffic details.
 
  Amos
 


___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Amos Jeffries]

On 20/03/2015 1:07 p.m., Dan Charlesworth wrote:
 Well I got 3.5.2 into production for a few hours and Bad Things happened:
 
 *1) A hefty performance hit*
 Load average was maybe a tad higher but CPU. memory and I/O were about the 
 same. 
 However the system seemed to top out at around 40 requests per second (on a 
 client that usually hits 100—150 rps) and squid became very slow to respond 
 to 
 squidclient requests:
 [root@proxy-LS5 ~]# time squidclient -p 8080 mgr:utilization | grep 
 client_http.requests
 client_http.requests = 40.965955/sec
 client_http.requests = 41.168528/sec
 client_http.requests = 42.111847/sec
 client_http.requests = 166646
 
 real0m7.163s
 user0m0.002s
 sys0m0.006s
 
 *2) Lots of Segment Violations*
 These obviously suck. Backtrace attached.

Still with symbols erased so I cant see what is going on :-(

(HackXBack is having a very similar set of issues, but on different
traffic)

 
 Just cannot win. Is it possible these two issues are due to the patch for 
 #4206?
 

Doubtful, 4206 patch was only affecting the tunnel.cc code handling for
CONNECT payloads. Your assertion is in client_side.cc

FWIW, the sponsor of that patch is happily running a 3.5.2 snapshot in a
very busy production system (~975 RPS at peak) with no problems reported
to me since it went live.
 We did find and fix a few of the other bugs that are patched in the
snapshot to get it to that state though. So if you are having problems
with the basic 3.5.2 bundle the latest snapshot may be worth a try as well.
There are also some patches in squid-4 that I'm due to be backporting in
the next few hrs related to worker security permissions (being denied
when they should be allowed), and PID fiel issues

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

Well I got 3.5.2 into production for a few hours and Bad Things happened:1) A hefty performance hitLoad average was maybe a tad higher but CPU. memory and I/O were about the same. However the system seemed to top out at around 40 requests per second (on a client that usually hits 100—150 rps) and squid became very slow to respond to squidclient requests:[root@proxy-LS5 ~]# time squidclient -p 8080 mgr:utilization | grep client_http.requestsclient_http.requests = 40.965955/secclient_http.requests = 41.168528/secclient_http.requests = 42.111847/secclient_http.requests = 166646real	0m7.163suser	0m0.002ssys	0m0.006s2) Lots of Segment ViolationsThese obviously suck. Backtrace attached.Just cannot win. Is it possible these two issues are due to the patch for #4206?bt full
#0  0x00397e232625 in ?? ()
No symbol table info available.
#1  0x00397e233e05 in ?? ()
No symbol table info available.
#2  0x00bb88a8 in queried_keys ()
No symbol table info available.
#3  0x00bb88b0 in queried_keys ()
No symbol table info available.
#4  0x0039864f32c0 in ?? ()
No symbol table info available.
#5  0x0059000b in operator std::char_traitschar  (this=0x2f89f30) 
at /usr/include/c++/4.4.7/ostream:510
No locals.
#6  FileMap::grow (this=0x2f89f30) at filemap.cc:75
_dbo = @0x8d01b90
old_sz = 0
old_map = 0x8bbb9e0
__FUNCTION__ = grow
#7  0x0002 in ?? ()
No symbol table info available.
#8  0x3ffd091c087442c8 in ?? ()
No symbol table info available.
#9  0x00bb91e0 in queried_keys ()
No symbol table info available.
#10 0x0001 in ?? ()
No symbol table info available.
#11 0x000c6e84 in ?? ()
No symbol table info available.
#12 0x0002 in ?? ()
No symbol table info available.
#13 0x4135 in ?? ()
No symbol table info available.
#14 0x0020 in ?? ()
No symbol table info available.
#15 0x in ?? ()
No symbol table info available.
On 16 Mar 2015, at 6:18 pm, Amos Jeffries squ...@treenet.co.nz wrote:On 16/03/2015 7:16 p.m., Dan Charlesworth wrote:Hey again Amos -Unfortunately the patch for #4206 won’t apply to squid-3.4.12. I was going to try creating a new one but couldn’t find an equivalentline in client_side.cc for that version.I guess the #4206 issue doesn’t apply to v3.4.x after all?Correct. Oh well.[Not a C programmer]Thanks for your time today.P.S. I'd love to upgrade to v3.5 but I'm waiting for somebody smarter than me to take the lead on a CentOS 6 RPM SPEC file.Eliezer to the rescue ;-)http://wiki.squid-cache.org/KnowledgeBase/CentOS#Squid-3.5Amos___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[johnzeng]

   Hello Dan:

i used 3.5.2 just now , i worried 3.5.3 isn't 
very stable too ,


i use 2.7stable 9 ago ,  and you ?

   if version is 3.xxx , which version is stablest 
until now .



   Best Regard

于 2015年03月20日 08:07, Dan Charlesworth 写道:

Well I got 3.5.2 into production for a few hours and Bad Things happened:

*1) A hefty performance hit*
Load average was maybe a tad higher but CPU. memory and I/O were about 
the same. However the system seemed to top out at around 40 requests 
per second (on a client that usually hits 100—150 rps) and squid 
became very slow to respond to squidclient requests:
[root@proxy-LS5 ~]# time squidclient -p 8080 mgr:utilization | grep 
client_http.requests

client_http.requests = 40.965955/sec
client_http.requests = 41.168528/sec
client_http.requests = 42.111847/sec
client_http.requests = 166646

real0m7.163s
user0m0.002s
sys0m0.006s

*2) Lots of Segment Violations*
These obviously suck. Backtrace attached.

Just cannot win. Is it possible these two issues are due to the patch 
for #4206?





On 16 Mar 2015, at 6:18 pm, Amos Jeffries squ...@treenet.co.nz 
mailto:squ...@treenet.co.nz wrote:


On 16/03/2015 7:16 p.m., Dan Charlesworth wrote:

Hey again Amos -

Unfortunately the patch for #4206 won’t apply to squid-3.4.12. I was 
going to try creating a new one but couldn’t find an equivalent line 
in client_side.cc for that version.


I guess the #4206 issue doesn’t apply to v3.4.x after all?


Correct. Oh well.




[Not a C programmer]

Thanks for your time today.

P.S. I'd love to upgrade to v3.5 but I'm waiting for somebody 
smarter than me to take the lead on a CentOS 6 RPM SPEC file.


Eliezer to the rescue ;-)
http://wiki.squid-cache.org/KnowledgeBase/CentOS#Squid-3.5


Amos





___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

John -

For us the 3.4 series is definitely the stablest.

I was hoping 3.5.2 + plus a patch would avoid the error in this thread’s 
subject—and it might have done—but it introduced two other major problems (for 
us).

 On 20 Mar 2015, at 2:29 pm, johnzeng johnzeng2...@yahoo.com wrote:
 
 
 Hello Dan:
 
 i used squid 2.7stable9 ago ,and i worried whether squid 
 3.5.2 is stablest for us until now too .
 
 and you ?
 
 Do you think Whether version is stablest at squid 3.xxx  ?
 
 
 
 
 
 
 
 Well I got 3.5.2 into production for a few hours and Bad Things happened:
 
 *1) A hefty performance hit*
 Load average was maybe a tad higher but CPU. memory and I/O were about the 
 same. However the system seemed to top out at around 40 requests per second 
 (on a client that usually hits 100—150 rps) and squid became very slow to 
 respond to squidclient requests:
 [root@proxy-LS5 ~]# time squidclient -p 8080 mgr:utilization | grep 
 client_http.requests
 client_http.requests = 40.965955/sec
 client_http.requests = 41.168528/sec
 client_http.requests = 42.111847/sec
 client_http.requests = 166646
 
 real0m7.163s
 user0m0.002s
 sys0m0.006s
 
 *2) Lots of Segment Violations*
 These obviously suck. Backtrace attached.
 
 Just cannot win. Is it possible these two issues are due to the patch for 
 #4206?
 
 
 
 
 On 16 Mar 2015, at 6:18 pm, Amos Jeffries squ...@treenet.co.nz 
 mailto:squ...@treenet.co.nz wrote:
 
 On 16/03/2015 7:16 p.m., Dan Charlesworth wrote:
 Hey again Amos -
 
 Unfortunately the patch for #4206 won’t apply to squid-3.4.12. I was going 
 to try creating a new one but couldn’t find an equivalent line in 
 client_side.cc for that version.
 
 I guess the #4206 issue doesn’t apply to v3.4.x after all?
 
 Correct. Oh well.
 
 
 
 [Not a C programmer]
 
 Thanks for your time today.
 
 P.S. I'd love to upgrade to v3.5 but I'm waiting for somebody smarter than 
 me to take the lead on a CentOS 6 RPM SPEC file.
 
 Eliezer to the rescue ;-)
 http://wiki.squid-cache.org/KnowledgeBase/CentOS#Squid-3.5
 
 
 Amos
 
 
 
 
 ___
 squid-users mailing list
 squid-users@lists.squid-cache.org
 http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[johnzeng]

Hello Dan:

 i used squid 2.7stable9 ago ,and i worried whether 
squid 3.5.2 is stablest for us until now too .


 and you ?

 Do you think Whether version is stablest at squid 
3.xxx  ?









Well I got 3.5.2 into production for a few hours and Bad Things happened:

*1) A hefty performance hit*
Load average was maybe a tad higher but CPU. memory and I/O were about 
the same. However the system seemed to top out at around 40 requests 
per second (on a client that usually hits 100—150 rps) and squid 
became very slow to respond to squidclient requests:
[root@proxy-LS5 ~]# time squidclient -p 8080 mgr:utilization | grep 
client_http.requests

client_http.requests = 40.965955/sec
client_http.requests = 41.168528/sec
client_http.requests = 42.111847/sec
client_http.requests = 166646

real0m7.163s
user0m0.002s
sys0m0.006s

*2) Lots of Segment Violations*
These obviously suck. Backtrace attached.

Just cannot win. Is it possible these two issues are due to the patch 
for #4206?





On 16 Mar 2015, at 6:18 pm, Amos Jeffries squ...@treenet.co.nz 
mailto:squ...@treenet.co.nz wrote:


On 16/03/2015 7:16 p.m., Dan Charlesworth wrote:

Hey again Amos -

Unfortunately the patch for #4206 won’t apply to squid-3.4.12. I was 
going to try creating a new one but couldn’t find an equivalent line 
in client_side.cc for that version.


I guess the #4206 issue doesn’t apply to v3.4.x after all?


Correct. Oh well.




[Not a C programmer]

Thanks for your time today.

P.S. I'd love to upgrade to v3.5 but I'm waiting for somebody 
smarter than me to take the lead on a CentOS 6 RPM SPEC file.


Eliezer to the rescue ;-)
http://wiki.squid-cache.org/KnowledgeBase/CentOS#Squid-3.5


Amos





___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Amos Jeffries]

On 27/02/2015 12:25 p.m., Dan Charlesworth wrote:
 Alright I got abrtd on board, finally.
 
 Here’s a a backtrace from this morning (bt and bt full versions included 
 separately):
 

Wonderful.

Can you get a print from frame 3 please of which of the connIsUsable()
checks if failing?


gdb commands:
 frame 3
 print http-getConn()
 print cbdataReferenceValid(http-getConn())
 print Comm::IsConnOpen(http-getConn()-clientConnection)

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

Alright I got abrtd on board, finally.Here’s a a backtrace from this morning (bt and bt full versions included separately):#0  0x00397e232625 in raise (sig=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00397e233e05 in abort () at abort.c:92
#2  0x005656ef in xassert (msg=0x80e183 
connIsUsable(http-getConn()), file=0x80d070 client_side.cc, line=1515) at 
debug.cc:566
#3  0x0053e2cf in clientSocketRecipient (node=0x49e03568, 
http=0x46f9fdf8, rep=0x6030a6a0, receivedData=...) at client_side.cc:1515
#4  0x00546207 in clientReplyContext::processReplyAccessResult 
(this=0x46fa2bf8, accessAllowed=value optimized out) at 
client_side_reply.cc:2058
#5  0x00546643 in clientReplyContext::ProcessReplyAccessResult (rv=..., 
voidMe=value optimized out) at client_side_reply.cc:1961
#6  0x006ec9db in ACLChecklist::checkCallback (this=0x31210228, 
answer=value optimized out) at Checklist.cc:161
#7  0x00588daf in externalAclHandleReply (data=value optimized out, 
reply=value optimized out) at external_acl.cc:1427
#8  0x005bd62a in helperReturnBuffer (conn=value optimized out, 
buf=value optimized out, len=value optimized out, flag=value optimized 
out, 
xerrno=value optimized out, data=0x4fadea8) at helper.cc:858
#9  helperHandleRead (conn=value optimized out, buf=value optimized out, 
len=value optimized out, flag=value optimized out, xerrno=value optimized 
out, 
data=0x4fadea8) at helper.cc:951
#10 0x006efde6 in AsyncCall::make (this=0x6031d880) at AsyncCall.cc:32
#11 0x006f2ea2 in AsyncCallQueue::fireNext (this=value optimized out) 
at AsyncCallQueue.cc:52
#12 0x006f31f0 in AsyncCallQueue::fire (this=0x1a45f30) at 
AsyncCallQueue.cc:38
#13 0x00584a34 in EventLoop::runOnce (this=0x7cf244c0) at 
EventLoop.cc:135
#14 0x00584b88 in EventLoop::run (this=0x7cf244c0) at 
EventLoop.cc:99
#15 0x00604918 in SquidMain (argc=value optimized out, 
argv=0x7cf246b8) at main.cc:1528
#16 0x006052a8 in SquidMainSafe (argc=value optimized out, 
argv=value optimized out) at main.cc:1260
#17 main (argc=value optimized out, argv=value optimized out) at 
main.cc:1252
#0  0x00397e232625 in raise (sig=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt full
#0  0x00397e232625 in raise (sig=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:64
resultvar = 0
pid = value optimized out
selftid = 22170
#1  0x00397e233e05 in abort () at abort.c:92
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x735573496e6e6f63, 
sa_sigaction = 0x735573496e6e6f63}, sa_mask = {__val = {8391446528407659105, 
95, 246932897408, 
  1515, 11643056, 1613801120, 2116527793, 70, 779769992, 779769992, 
779769904, 779769992, 247066473152, 140737437122128, 140737437122096, 
1613801120}}, 
  sa_flags = -2044089756, sa_restorer = 0xb1a8d0 Debug::CurrentDebug}
sigs = {__val = {32, 0 repeats 15 times}}
#2  0x005656ef in xassert (msg=0x80e183 
connIsUsable(http-getConn()), file=0x80d070 client_side.cc, line=1515) at 
debug.cc:566
__FUNCTION__ = xassert
#3  0x0053e2cf in clientSocketRecipient (node=0x49e03568, 
http=0x46f9fdf8, rep=0x6030a6a0, receivedData=...) at client_side.cc:1515
context = {p_ = 0x46fa1b48}
mustSendLastChunk = value optimized out
#4  0x00546207 in clientReplyContext::processReplyAccessResult 
(this=0x46fa2bf8, accessAllowed=value optimized out) at 
client_side_reply.cc:2058
__FUNCTION__ = processReplyAccessResult
localTempBuffer = {flags = {error = value optimized out}, length = 0, 
offset = value optimized out, data = 0x46fa1cb4 }
buf = 0x46fa1b68 HTTP/1.1 200 OK\r\nServer: Apache\r\nETag: 
\7d861e702caf2102333f5e730b15fa7d:1424978619\\r\nLast-Modified: Thu, 26 Feb 
2015 19:00:47 GMT\r\nAccept-Ranges: bytes\r\nContent-Length: 
18206\r\nContent-Type: image/png...
body_buf = value optimized out
body_size = value optimized out
#5  0x00546643 in clientReplyContext::ProcessReplyAccessResult (rv=..., 
voidMe=value optimized out) at client_side_reply.cc:1961
me = value optimized out
#6  0x006ec9db in ACLChecklist::checkCallback (this=0x31210228, 
answer=value optimized out) at Checklist.cc:161
callback_ = 0x546630 
clientReplyContext::ProcessReplyAccessResult(allow_t, void*)
cbdata_ = 0x46fa2bf8
__FUNCTION__ = checkCallback
#7  0x00588daf in externalAclHandleReply (data=value optimized out, 
reply=value optimized out) at external_acl.cc:1427
cbdata = 0x31210228
state = 0x4cc1d078
__FUNCTION__ = externalAclHandleReply
next = value optimized out
entryData = {result = {code = ACCESS_DENIED, kind = 0}, notes = {Lock 
= {_vptr.Lock = 0xaedb80, count_ = 0}, _vptr.NotePairs = 0xaedb58, 

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[dan]

By the way, I think Eliezer suggested I should file a bug report. There is in 
fact one already filed here:
http://bugs.squid-cache.org/show_bug.cgi?id=3930




Which brings me to my next question ...



Amos, is it possible to sponsor bug fixes?

On Tue, Feb 24, 2015 at 2:47 PM, null d...@getbusi.com wrote:

 This is kind of off-topic but on one of our deployments this crash is now 
 consistently deadlocking squid whenever it occurs rather than just ending the 
 process. Meaning that is can’t be restarted by any means except kill -9, 
 which obviously a huge disruption to hundreds of clients and incredibly 
 frustrating for the sysadmin.
 Nothing has really changed in the configuration since this deadlocking 
 started happening  but I’ve noticed when that there’s no longer anything in 
 /var/log/messages from abrtd etc. like there usually would be.
 I have a very similar deployment where this still crashes “cleanly”.
 Both on CentOS 6.6 and Squid 3.4.12.
 Anyone have a clue what might cause this “deadlocking” type behaviour after 
 an “assertion failed crash?
 On Fri, Feb 20, 2015 at 5:23 PM, Amos Jeffries squ...@treenet.co.nz
 wrote:
 On 20/02/2015 7:15 p.m., Dan Charlesworth wrote:
 Thanks Amos -
 
 So then it more than likely is related to our external ACLs that deal with 
 the HTTP response?
 
 I think they may be making the issue more noticable by slowing down the
 request processing. But Squid should not be getting into that state
 either way.
 Amos___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Amos Jeffries]

On 2015-02-25 15:11, d...@getbusi.com wrote:

By the way, I think Eliezer suggested I should file a bug report.
There is in fact one already filed here:
http://bugs.squid-cache.org/show_bug.cgi?id=3930


Which brings me to my next question ...



Amos, is it possible to sponsor bug fixes?


Of course. Though be aware that if its an outstanding bug it means we 
are having difficulty resolving it for some reason so we can't really 
provide any guarantees about what it will take to fix.


So far the issue seems to be a lack of information, with a stack trace 
only coming in recently and the build and config details having come 
from someone other than the one with stack trace. Ideally we need all 
the details from a single proxy instance encountering the crash.


Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[dan]

This is kind of off-topic but on one of our deployments this crash is now 
consistently deadlocking squid whenever it occurs rather than just ending the 
process. Meaning that is can’t be restarted by any means except kill -9, which 
obviously a huge disruption to hundreds of clients and incredibly frustrating 
for the sysadmin.




Nothing has really changed in the configuration since this deadlocking started 
happening  but I’ve noticed when that there’s no longer anything in 
/var/log/messages from abrtd etc. like there usually would be.




I have a very similar deployment where this still crashes “cleanly”.




Both on CentOS 6.6 and Squid 3.4.12.




Anyone have a clue what might cause this “deadlocking” type behaviour after an 
“assertion failed crash?

On Fri, Feb 20, 2015 at 5:23 PM, Amos Jeffries squ...@treenet.co.nz
wrote:

 On 20/02/2015 7:15 p.m., Dan Charlesworth wrote:
 Thanks Amos -
 
 So then it more than likely is related to our external ACLs that deal with 
 the HTTP response?
 
 I think they may be making the issue more noticable by slowing down the
 request processing. But Squid should not be getting into that state
 either way.
 Amos___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Amos Jeffries]

On 20/02/2015 5:46 p.m., Eliezer Croitoru wrote:
 Hey Dan,
 
 The basic rule of thumb in programming lands is script vs compiled code.
 Where compiled code can be considered very reliable and in most cases
 tested much more then scripts.
 I am fearing that there is some race between all sorts of things on
 runtime which might lead to this failed test.
 
 There are couple possibilities that can cause the issue you are writing
 about.
 From the compiled side of the code the main suspect is that the
 connection got into a non usable state before squid could do something
 else.
 I have not seen yet the source code for connIsUsable but if you wish I
 can try and look at the function\method\call\code source and start a
 basic lookup to understand the issue a bit more.

Its a simple test to check to ensure the client connection is open when
writing some response data to it.

Something has earlier cause client connection closure, and something
else earlier has failed to cleanup the state or check the state was sane
before getting to the point of assertion.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

Thanks Eliezer …

We've only ever used `kill` as very last resort when the squid process wouldn’t 
respond to anything else.

Anyway, I think I missed what led you to think the crash is related to the 
reply_body_max_size rules' external ACL as opposed to the many others we define?

That would certainly narrow it down a lot further than before.

Cheers
Dan

 On 20 Feb 2015, at 2:57 pm, Eliezer Croitoru elie...@ngtech.co.il wrote:
 
 Hey Dan,
 
 I am not the best at reading squid long debug output and it is needed in 
 order to understand the path that the request is traveling between the ACLs 
 and helper to determine if the issue is since the connection is unusable 
 because of a helper or because of another reason.
 
 And so from what you describe I assume that the needed helper\external ACL is 
 a fake one so a python helper is out of the question for such a purpose.
 The fact that it's crashing describes some kind of failure from the 
 combination of something.
 In order to test if the issue is because of the helper or something related 
 to the existence of this specific helper for the reply body max try to 
 disable this helper and use only the basic limit, while it will force you to 
 not show a nice deny info page it will prevent the some of the issues from 
 happening.
 
 From stability point of view running all these kill -9 what so ever is a very 
 wrong approach.
 The crashes else then causing down time causing a much deeper issue.
 Assuming that the users transactions are important these crashes are damaging 
 in many cases even more then any down time.
 I know that some admins do not agree with my approach but a stable service is 
 one of the basic fundamentals for success and happiness!!
 
 I must admit that there are cases which a kill -9 can help but it has it's 
 price.
 
 Just asking loudly from both CEO + SYSADMIN + CLIENTS + others:
 What would you prefer?
 - stability based a good product
 - stability based on patches
 - stability based on human resources recruitment's
 - stability based on some unclosed known bugs
 - stability based on 1k programmers work
 - stability based on protocol compatibility
 
 
 And I must stop here with the list since the above list can become very long 
 and which will prove that humans can look at the same picture and see many 
 different things.
 
 Eliezer
 
 * I am almost sure that you may use a fake acl that will match all requests 
 instead of using an external_acl helper that will help you to select the 
 100MB limit.
 
 On 20/02/2015 05:34, Dan Charlesworth wrote:
 Installed v3.4.12 and almost went a whole day without this crash.
 Ended up rearing its head during a spike in traffic after lunch. Seems
 to be more prone to occurring when the HTTP requests per second
 reaches about 100.
 
 I have a script running that runs a squid reload whenever this crash
 occurs and that seems to limit the impact (downtime) to a few
 seconds—but occasionally Squid seems to get deadlocked by the crash
 and needs to be killed (sometimes with -9) before it can be restarted.
 
 In lieu of being able to diagnose and fix this, does anyone have any
 other creative ideas as to limiting its impact?
 
 Thanks
 Dan
 
 
 On 12 February 2015 at 09:51, Dan Charlesworth d...@getbusi.com wrote:
 Hey Eliezer
 
 With the response_size_100 ACL definition:
 - 100 tells the external ACL the limit in MB
 - 192.168.0.10 tells the external ACL the squid IP
 
 I think one or both of these is only needed to build the deny page. You 
 can’t use deny_info with reply_body_max_size so we had to customise the 
 ERR_TOO_BIG source to do a redirect to our own page.
 
 The http_access allow line is because result caching cannot alter the 
 EXT_LOG for fast ACLs as cache lookups include the EXT_LOG, so we need to 
 check the result twice to alter the EXT_LOG and then have the result cached 
 against the altered EXT_LOG.
 
 Cheers
 Dan
 
 On 11 Feb 2015, at 11:09 pm, Eliezer Croitoru elie...@ngtech.co.il wrote:
 
 Hey Dan,
 
 First I must admit that this squid.conf is quite complicated but kind of 
 self explanatory.
 
 I have tried to understand the next lines:
 # File size (download) restrictions
 acl response_size_100 external response_size_type 100 192.168.0.10
 http_access allow response_size_100 response_size_100
 reply_body_max_size 100 MB response_size_100
 
 But I am unsure how it works with external_acl exactly.
 If you wish to deny 100MB size files you should have only one rule for the 
 reply body max size, what are the others for exactly?
 
 Eliezer
 
 * I might missing some concepts some sorry in advance.
 
 On 11/02/2015 00:30, Dan Charlesworth wrote:
 Hi Eliezer
 
 Took a while to get this up—sorry about that. Here’s an example of a 
 production config of ours (with some confidential stuff necessarily taken 
 out/edited):
 https://gist.github.com/djch/92cf0b04afbd7917  
 https://gist.github.com/djch/92cf0b04afbd7917
 
 Let me know if there’s any other info I can provide 

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Eliezer Croitoru]

Hey Dan,

The basic rule of thumb in programming lands is script vs compiled code.
Where compiled code can be considered very reliable and in most cases 
tested much more then scripts.
I am fearing that there is some race between all sorts of things on 
runtime which might lead to this failed test.


There are couple possibilities that can cause the issue you are writing 
about.
From the compiled side of the code the main suspect is that the 
connection got into a non usable state before squid could do something 
else.
I have not seen yet the source code for connIsUsable but if you wish I 
can try and look at the function\method\call\code source and start a 
basic lookup to understand the issue a bit more.
I do not remember if you have opened a bug report yet but I think this 
issue needs a bug report.


Take your time to write a brief bug report at:
http://bugs.squid-cache.org/index.cgi

While referring to this thread.
If you are up for the task then maybe you would be able to provide some 
more information based on the wiki:

http://wiki.squid-cache.org/SquidFaq/BugReporting

Thanks,
Eliezer

On 20/02/2015 06:06, Dan Charlesworth wrote:

Thanks Eliezer …

We've only ever used `kill` as very last resort when the squid process wouldn’t 
respond to anything else.

Anyway, I think I missed what led you to think the crash is related to the 
reply_body_max_size rules' external ACL as opposed to the many others we define?

That would certainly narrow it down a lot further than before.

Cheers
Dan



___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Amos Jeffries]

On 20/02/2015 7:15 p.m., Dan Charlesworth wrote:
 Thanks Amos -
 
 So then it more than likely is related to our external ACLs that deal with 
 the HTTP response?
 

I think they may be making the issue more noticable by slowing down the
request processing. But Squid should not be getting into that state
either way.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

Thanks Amos -

So then it more than likely is related to our external ACLs that deal with the 
HTTP response?

 On 20 Feb 2015, at 5:06 pm, Amos Jeffries squ...@treenet.co.nz wrote:
 
 On 20/02/2015 5:46 p.m., Eliezer Croitoru wrote:
 Hey Dan,
 
 The basic rule of thumb in programming lands is script vs compiled code.
 Where compiled code can be considered very reliable and in most cases
 tested much more then scripts.
 I am fearing that there is some race between all sorts of things on
 runtime which might lead to this failed test.
 
 There are couple possibilities that can cause the issue you are writing
 about.
 From the compiled side of the code the main suspect is that the
 connection got into a non usable state before squid could do something
 else.
 I have not seen yet the source code for connIsUsable but if you wish I
 can try and look at the function\method\call\code source and start a
 basic lookup to understand the issue a bit more.
 
 Its a simple test to check to ensure the client connection is open when
 writing some response data to it.
 
 Something has earlier cause client connection closure, and something
 else earlier has failed to cleanup the state or check the state was sane
 before getting to the point of assertion.
 
 Amos
 
 ___
 squid-users mailing list
 squid-users@lists.squid-cache.org
 http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Eliezer Croitoru]

Hey Dan,

I am not the best at reading squid long debug output and it is needed in 
order to understand the path that the request is traveling between the 
ACLs and helper to determine if the issue is since the connection is 
unusable because of a helper or because of another reason.


And so from what you describe I assume that the needed helper\external 
ACL is a fake one so a python helper is out of the question for such a 
purpose.
The fact that it's crashing describes some kind of failure from the 
combination of something.
In order to test if the issue is because of the helper or something 
related to the existence of this specific helper for the reply body max 
try to disable this helper and use only the basic limit, while it will 
force you to not show a nice deny info page it will prevent the some of 
the issues from happening.


From stability point of view running all these kill -9 what so ever is 
a very wrong approach.

The crashes else then causing down time causing a much deeper issue.
Assuming that the users transactions are important these crashes are 
damaging in many cases even more then any down time.
I know that some admins do not agree with my approach but a stable 
service is one of the basic fundamentals for success and happiness!!


I must admit that there are cases which a kill -9 can help but it has 
it's price.


Just asking loudly from both CEO + SYSADMIN + CLIENTS + others:
What would you prefer?
- stability based a good product
- stability based on patches
- stability based on human resources recruitment's
- stability based on some unclosed known bugs
- stability based on 1k programmers work
- stability based on protocol compatibility


And I must stop here with the list since the above list can become very 
long and which will prove that humans can look at the same picture and 
see many different things.


Eliezer

* I am almost sure that you may use a fake acl that will match all 
requests instead of using an external_acl helper that will help you to 
select the 100MB limit.


On 20/02/2015 05:34, Dan Charlesworth wrote:

Installed v3.4.12 and almost went a whole day without this crash.
Ended up rearing its head during a spike in traffic after lunch. Seems
to be more prone to occurring when the HTTP requests per second
reaches about 100.

I have a script running that runs a squid reload whenever this crash
occurs and that seems to limit the impact (downtime) to a few
seconds—but occasionally Squid seems to get deadlocked by the crash
and needs to be killed (sometimes with -9) before it can be restarted.

In lieu of being able to diagnose and fix this, does anyone have any
other creative ideas as to limiting its impact?

Thanks
Dan


On 12 February 2015 at 09:51, Dan Charlesworth d...@getbusi.com wrote:

Hey Eliezer

With the response_size_100 ACL definition:
- 100 tells the external ACL the limit in MB
- 192.168.0.10 tells the external ACL the squid IP

I think one or both of these is only needed to build the deny page. You can’t 
use deny_info with reply_body_max_size so we had to customise the ERR_TOO_BIG 
source to do a redirect to our own page.

The http_access allow line is because result caching cannot alter the EXT_LOG 
for fast ACLs as cache lookups include the EXT_LOG, so we need to check the 
result twice to alter the EXT_LOG and then have the result cached against the 
altered EXT_LOG.

Cheers
Dan


On 11 Feb 2015, at 11:09 pm, Eliezer Croitoru elie...@ngtech.co.il wrote:

Hey Dan,

First I must admit that this squid.conf is quite complicated but kind of self 
explanatory.

I have tried to understand the next lines:
# File size (download) restrictions
acl response_size_100 external response_size_type 100 192.168.0.10
http_access allow response_size_100 response_size_100
reply_body_max_size 100 MB response_size_100

But I am unsure how it works with external_acl exactly.
If you wish to deny 100MB size files you should have only one rule for the 
reply body max size, what are the others for exactly?

Eliezer

* I might missing some concepts some sorry in advance.

On 11/02/2015 00:30, Dan Charlesworth wrote:

Hi Eliezer

Took a while to get this up—sorry about that. Here’s an example of a production 
config of ours (with some confidential stuff necessarily taken out/edited):
https://gist.github.com/djch/92cf0b04afbd7917  
https://gist.github.com/djch/92cf0b04afbd7917

Let me know if there’s any other info I can provide that might point towards 
the cause of this crash.

And thanks again for taking a look.








___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

Installed v3.4.12 and almost went a whole day without this crash.
Ended up rearing its head during a spike in traffic after lunch. Seems
to be more prone to occurring when the HTTP requests per second
reaches about 100.

I have a script running that runs a squid reload whenever this crash
occurs and that seems to limit the impact (downtime) to a few
seconds—but occasionally Squid seems to get deadlocked by the crash
and needs to be killed (sometimes with -9) before it can be restarted.

In lieu of being able to diagnose and fix this, does anyone have any
other creative ideas as to limiting its impact?

Thanks
Dan


On 12 February 2015 at 09:51, Dan Charlesworth d...@getbusi.com wrote:
 Hey Eliezer

 With the response_size_100 ACL definition:
 - 100 tells the external ACL the limit in MB
 - 192.168.0.10 tells the external ACL the squid IP

 I think one or both of these is only needed to build the deny page. You can’t 
 use deny_info with reply_body_max_size so we had to customise the ERR_TOO_BIG 
 source to do a redirect to our own page.

 The http_access allow line is because result caching cannot alter the EXT_LOG 
 for fast ACLs as cache lookups include the EXT_LOG, so we need to check the 
 result twice to alter the EXT_LOG and then have the result cached against the 
 altered EXT_LOG.

 Cheers
 Dan

 On 11 Feb 2015, at 11:09 pm, Eliezer Croitoru elie...@ngtech.co.il wrote:

 Hey Dan,

 First I must admit that this squid.conf is quite complicated but kind of 
 self explanatory.

 I have tried to understand the next lines:
 # File size (download) restrictions
 acl response_size_100 external response_size_type 100 192.168.0.10
 http_access allow response_size_100 response_size_100
 reply_body_max_size 100 MB response_size_100

 But I am unsure how it works with external_acl exactly.
 If you wish to deny 100MB size files you should have only one rule for the 
 reply body max size, what are the others for exactly?

 Eliezer

 * I might missing some concepts some sorry in advance.

 On 11/02/2015 00:30, Dan Charlesworth wrote:
 Hi Eliezer

 Took a while to get this up—sorry about that. Here’s an example of a 
 production config of ours (with some confidential stuff necessarily taken 
 out/edited):
 https://gist.github.com/djch/92cf0b04afbd7917  
 https://gist.github.com/djch/92cf0b04afbd7917

 Let me know if there’s any other info I can provide that might point 
 towards the cause of this crash.

 And thanks again for taking a look.



___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Eliezer Croitoru]

Hey Dan,

First I must admit that this squid.conf is quite complicated but kind of 
self explanatory.


I have tried to understand the next lines:
# File size (download) restrictions
acl response_size_100 external response_size_type 100 192.168.0.10
http_access allow response_size_100 response_size_100
reply_body_max_size 100 MB response_size_100

But I am unsure how it works with external_acl exactly.
If you wish to deny 100MB size files you should have only one rule for 
the reply body max size, what are the others for exactly?


Eliezer

* I might missing some concepts some sorry in advance.

On 11/02/2015 00:30, Dan Charlesworth wrote:

Hi Eliezer

Took a while to get this up—sorry about that. Here’s an example of a production 
config of ours (with some confidential stuff necessarily taken out/edited):
https://gist.github.com/djch/92cf0b04afbd7917  
https://gist.github.com/djch/92cf0b04afbd7917

Let me know if there’s any other info I can provide that might point towards 
the cause of this crash.

And thanks again for taking a look.



___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

Hi Eliezer

Took a while to get this up—sorry about that. Here’s an example of a production 
config of ours (with some confidential stuff necessarily taken out/edited):
https://gist.github.com/djch/92cf0b04afbd7917 
https://gist.github.com/djch/92cf0b04afbd7917

Let me know if there’s any other info I can provide that might point towards 
the cause of this crash.

And thanks again for taking a look.

 On 3 Feb 2015, at 2:49 pm, Dan Charlesworth d...@getbusi.com wrote:
 
 Hi Eliezer
 
 Thanks for paying attention, as always. I’m working on getting an 
 (appropriately censored) example of our squid.conf up for your perusal.
 
 In the mean time I just wanted to point out that when this crash occurs some 
 of the most busy external_acl_types appear to crash too. Though the exact 
 ones seems to vary a bit between occurrences:
 
 2015/02/03 13:03:05 kid1| assertion failed: client_side.cc:1515: 
 connIsUsable(http-getConn())
 Traceback (most recent call last):
   File max_file_size_acl.pyo, line 76, in module
 IOError: [Errno 104] Connection reset by peer
 2015/02/03 13:04:01 kid1| Set Current Directory to /var/spool/squid
 Traceback (most recent call last):
   File set_finder_acl.pyo, line 94, in module
 IOError: [Errno 104] Connection reset by peer
 2015/02/03 13:04:01 kid1| Starting Squid Cache version 3.4.11 for 
 x86_64-redhat-linux-gnu...
 
 Those lines it’s pointing to in the Traceback are just the last line in each 
 ACL e.g. `line = sys.stdin.readline()`
 
 Cheers
 Dan
 
 On 2 Feb 2015, at 11:35 am, Eliezer Croitoru elie...@ngtech.co.il 
 mailto:elie...@ngtech.co.il wrote:
 
 Hey Dan,
 
 Just to get around the environment, can you share your squid.conf?(censuring 
 confidential data)
 
 Thanks,
 Eliezer
 
 On 02/02/2015 01:14, Dan Charlesworth wrote:
 Bumping this one for the new year 'cause I still don't understand squid
 traces and because it's still happening with v3.4.11.
 
 I would speculate that's it's something to do with the External ACLs
 (there's a bunch). Let me know if a more recent traceback (than those
 earlier in the thread) would help.
 
 
 ___
 squid-users mailing list
 squid-users@lists.squid-cache.org mailto:squid-users@lists.squid-cache.org
 http://lists.squid-cache.org/listinfo/squid-users
 

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

Hi Eliezer

Thanks for paying attention, as always. I’m working on getting an 
(appropriately censored) example of our squid.conf up for your perusal.

In the mean time I just wanted to point out that when this crash occurs some of 
the most busy external_acl_types appear to crash too. Though the exact ones 
seems to vary a bit between occurrences:

2015/02/03 13:03:05 kid1| assertion failed: client_side.cc:1515: 
connIsUsable(http-getConn())
Traceback (most recent call last):
  File max_file_size_acl.pyo, line 76, in module
IOError: [Errno 104] Connection reset by peer
2015/02/03 13:04:01 kid1| Set Current Directory to /var/spool/squid
Traceback (most recent call last):
  File set_finder_acl.pyo, line 94, in module
IOError: [Errno 104] Connection reset by peer
2015/02/03 13:04:01 kid1| Starting Squid Cache version 3.4.11 for 
x86_64-redhat-linux-gnu...

Those lines it’s pointing to in the Traceback are just the last line in each 
ACL e.g. `line = sys.stdin.readline()`

Cheers
Dan

 On 2 Feb 2015, at 11:35 am, Eliezer Croitoru elie...@ngtech.co.il wrote:
 
 Hey Dan,
 
 Just to get around the environment, can you share your squid.conf?(censuring 
 confidential data)
 
 Thanks,
 Eliezer
 
 On 02/02/2015 01:14, Dan Charlesworth wrote:
 Bumping this one for the new year 'cause I still don't understand squid
 traces and because it's still happening with v3.4.11.
 
 I would speculate that's it's something to do with the External ACLs
 (there's a bunch). Let me know if a more recent traceback (than those
 earlier in the thread) would help.
 
 
 ___
 squid-users mailing list
 squid-users@lists.squid-cache.org
 http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Dan Charlesworth]

Bumping this one for the new year 'cause I still don't understand squid
traces and because it's still happening with v3.4.11.

I would speculate that's it's something to do with the External ACLs
(there's a bunch). Let me know if a more recent traceback (than those
earlier in the thread) would help.

On 2 February 2015 at 10:14, Dan Charlesworth d...@getbusi.com wrote:

 Bumping this one for the new year 'cause I still don't understand squid
 traces and because it's still happening with v3.4.11.

 I would speculate that's it's something to do with the External ACLs
 (there's a bunch). Let me know if a more recent traceback (than those
 earlier in the thread) would help.

 On 13 November 2014 at 16:02, d...@getbusi.com wrote:

 Oh sure, sorry:

  Squid Cache: Version 3.4.8
 configure options:  '--build=x86_64-redhat-linux-gnu'
 '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu'
 '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
 '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
 '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64'
 '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib'
 '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr'
 '--libexecdir=/usr/lib64/squid' '--localstatedir=/var'
 '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid'
 '--with-logdir=$(localstatedir)/log/squid'
 '--with-pidfile=$(localstatedir)/run/squid.pid'
 '--disable-dependency-tracking' '--enable-follow-x-forwarded-for'
 '--enable-auth'
 '--enable-auth-basic=DB,LDAP,MSNT,MSNT-multi-domain,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,getpwnam'
 '--enable-auth-ntlm=smb_lm,fake'
 '--enable-auth-digest=file,LDAP,eDirectory'
 '--enable-auth-negotiate=kerberos,wrapper'
 '--enable-external-acl-helpers=wbinfo_group,kerberos_ldap_group,AD_group,session'
 '--enable-cache-digests' '--enable-cachemgr-hostname=localhost'
 '--enable-delay-pools' '--enable-epoll' '--enable-icap-client'
 '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-referer-log'
 '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl'
 '--enable-ssl-crtd' '--enable-storeio=aufs,diskd,ufs'
 '--enable-useragent-log' '--enable-wccpv2' '--enable-esi' '--with-aio'
 '--with-default-user=squid' '--with-filedescriptors=16384'
 '--with-maxfd=65535' '--with-dl' '--with-openssl' '--with-pthreads'
 '--with-included-ltdl' 'build_alias=x86_64-redhat-linux-gnu'
 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu'
 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
 -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'
 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
 -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC'
 'PKG_CONFIG_PATH=/usr/lib64/pkgconfig:/usr/share/pkgconfig'
 --enable-ltdl-convenience





 On Thu, Nov 13, 2014 at 4:01 PM, Amos Jeffries squ...@treenet.co.nz
 wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 7/11/2014 12:25 p.m., dan wrote:
  Bumping this with another backtrace. Happened at 16:05 this time,
  when the system was not very very busy.
 
  It’s causing squid to crash in such a way that I actually have to
  `kill -9` the process in order to get things restarted properly.
 
  Would really appreciate any feedback at all from anyone who can
  understand these back traces.


 Any hints, like what release version of Squid you are using?

 Amos

 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2.0.22 (MingW32)

 iQEcBAEBAgAGBQJUZDr6AAoJELJo5wb/XPRjxL8H/iQO8pG5twVV2lcXxlFEgLuE
 NR0c4ezkPOiwgM3iHzrVxcDtCRLHB2YrwT9GuapslmSkcTOP6sBKxekOHsZmWtlK
 Sd8A/jK6l/GXFqPpdUHjut6g1aEUwTfJxsRr2NxtMW2f7a91qyOE9f31WYKQq73m
 odjtt6ayc+yA2jMEfHaIHaqXhIzAxV21ipN8GH5CWhrfMfo6IxpP4326z8SMa1am
 6HXJQhTkt1qqV4jCjGdYQ4BkAZygBtsHNb2AKgJ5Wmb4OCsM4MZlbIiPmWqWWCfY
 8ccyLVvodfpPVtjCBgStcJTkWxamu6BaHNhy8qCV03zAa4faxqcYVwAvSA5Q2sg=
 =tHfy
 -END PGP SIGNATURE-
 ___
 squid-users mailing list
 squid-users@lists.squid-cache.org
 http://lists.squid-cache.org/listinfo/squid-users




___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Eliezer Croitoru]

Hey Dan,

Just to get around the environment, can you share your 
squid.conf?(censuring confidential data)


Thanks,
Eliezer

On 02/02/2015 01:14, Dan Charlesworth wrote:

Bumping this one for the new year 'cause I still don't understand squid
traces and because it's still happening with v3.4.11.

I would speculate that's it's something to do with the External ACLs
(there's a bunch). Let me know if a more recent traceback (than those
earlier in the thread) would help.



___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[Amos Jeffries]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 7/11/2014 12:25 p.m., dan wrote:
 Bumping this with another backtrace. Happened at 16:05 this time,
 when the system was not very very busy.
 
 It’s causing squid to crash in such a way that I actually have to
 `kill -9` the process in order to get things restarted properly.
 
 Would really appreciate any feedback at all from anyone who can
 understand these back traces.


Any hints, like what release version of Squid you are using?

Amos

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJUZDr6AAoJELJo5wb/XPRjxL8H/iQO8pG5twVV2lcXxlFEgLuE
NR0c4ezkPOiwgM3iHzrVxcDtCRLHB2YrwT9GuapslmSkcTOP6sBKxekOHsZmWtlK
Sd8A/jK6l/GXFqPpdUHjut6g1aEUwTfJxsRr2NxtMW2f7a91qyOE9f31WYKQq73m
odjtt6ayc+yA2jMEfHaIHaqXhIzAxV21ipN8GH5CWhrfMfo6IxpP4326z8SMa1am
6HXJQhTkt1qqV4jCjGdYQ4BkAZygBtsHNb2AKgJ5Wmb4OCsM4MZlbIiPmWqWWCfY
8ccyLVvodfpPVtjCBgStcJTkWxamu6BaHNhy8qCV03zAa4faxqcYVwAvSA5Q2sg=
=tHfy
-END PGP SIGNATURE-
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] assertion failed: client_side.cc:1515: connIsUsable(http-getConn())

[dan]

Oh sure, sorry:





Squid Cache: Version 3.4.8

configure options:  '--build=x86_64-redhat-linux-gnu' 
'--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' 
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' 
'--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' 
'--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' 
'--sharedstatedir=/var/lib' '--mandir=/usr/share/man' 
'--infodir=/usr/share/info' '--exec_prefix=/usr' 
'--libexecdir=/usr/lib64/squid' '--localstatedir=/var' 
'--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' 
'--with-logdir=$(localstatedir)/log/squid' 
'--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' 
'--enable-follow-x-forwarded-for' '--enable-auth' 
'--enable-auth-basic=DB,LDAP,MSNT,MSNT-multi-domain,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,getpwnam'
 '--enable-auth-ntlm=smb_lm,fake' '--enable-auth-digest=file,LDAP,eDirectory' 
'--enable-auth-negotiate=kerberos,wrapper' 
'--enable-external-acl-helpers=wbinfo_group,kerberos_ldap_group,AD_group,session'
 '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' 
'--enable-delay-pools' '--enable-epoll' '--enable-icap-client' 
'--enable-ident-lookups' '--enable-linux-netfilter' '--enable-referer-log' 
'--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' 
'--enable-ssl-crtd' '--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log' 
'--enable-wccpv2' '--enable-esi' '--with-aio' '--with-default-user=squid' 
'--with-filedescriptors=16384' '--with-maxfd=65535' '--with-dl' 
'--with-openssl' '--with-pthreads' '--with-included-ltdl' 
'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 
'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall 
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector 
--param=ssp-buffer-size=4 -m64 -mtune=generic' 'CXXFLAGS=-O2 -g -pipe -Wall 
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector 
--param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' 
'PKG_CONFIG_PATH=/usr/lib64/pkgconfig:/usr/share/pkgconfig' 
--enable-ltdl-convenience

On Thu, Nov 13, 2014 at 4:01 PM, Amos Jeffries squ...@treenet.co.nz
wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 On 7/11/2014 12:25 p.m., dan wrote:
 Bumping this with another backtrace. Happened at 16:05 this time,
 when the system was not very very busy.
 
 It’s causing squid to crash in such a way that I actually have to
 `kill -9` the process in order to get things restarted properly.
 
 Would really appreciate any feedback at all from anyone who can
 understand these back traces.
 Any hints, like what release version of Squid you are using?
 Amos
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2.0.22 (MingW32)
 iQEcBAEBAgAGBQJUZDr6AAoJELJo5wb/XPRjxL8H/iQO8pG5twVV2lcXxlFEgLuE
 NR0c4ezkPOiwgM3iHzrVxcDtCRLHB2YrwT9GuapslmSkcTOP6sBKxekOHsZmWtlK
 Sd8A/jK6l/GXFqPpdUHjut6g1aEUwTfJxsRr2NxtMW2f7a91qyOE9f31WYKQq73m
 odjtt6ayc+yA2jMEfHaIHaqXhIzAxV21ipN8GH5CWhrfMfo6IxpP4326z8SMa1am
 6HXJQhTkt1qqV4jCjGdYQ4BkAZygBtsHNb2AKgJ5Wmb4OCsM4MZlbIiPmWqWWCfY
 8ccyLVvodfpPVtjCBgStcJTkWxamu6BaHNhy8qCV03zAa4faxqcYVwAvSA5Q2sg=
 =tHfy
 -END PGP SIGNATURE-
 ___
 squid-users mailing list
 squid-users@lists.squid-cache.org
 http://lists.squid-cache.org/listinfo/squid-users___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users